DMOZ Import Mod

[f1gm3nt]

{non resolving url removed} - This version should get rid of the column errors. If it doesn't work right please post what other mods you have, if any, or any other kind of information.



DO NOT DOWNLOD THE FILE BELOW unless you want a useless outdated copy that does not work.. Please download the one above because it has fixed the column errors.

Can an admin please remove the file below?

[Ap0s7le]

Nah, don't post flames. No point, and Josh is doing everyone a favor so only Thank You's are accepted.

Thanks Josh, and I locked the other thread so people will start coming here instead.

-Casey

[f1gm3nt]

Thanks, this next one should be easier for everyone to use. I'm setting it up to where you want need to use phpMyAdmin....much... :roll:

[Ap0s7le]

Josh, if you need help yell.

Or just get a IM!! haha

The front end could be web based. Enter the Structure path you want, DB User/Pass and DB name.

It would go ahead, start the whole process. Log it's progress on screen as Cats / Subs / Links but log details in a TXT file.

Give the option to Append a PHPLD install, or a Fresh install. So it would know if the cats it's about to add are the only ones in there. If you opt for the append, it could then check the DB for existing fields for duplicate data.

Or something like that... I'm rambling... Going to lay down and watch a lil tv...

later

-Casey

[f1gm3nt]

Well, it has always checked to see if the categories and links are in the phpLD database so it knows which to add. If it's already there it doesn't add them

Right now I've got it set up to where you create the tables and edit the two files, class.dmoz.php ad class.sql.php.

class.dmoz.php you just change to the directory that you want.

You don't have to add the first link in the DB. The script does it for you now.

What kind of details would be placed in the txt file? It seems like a good idea but not sure what kind of info could be placed in there.

In the future I want it to mooch off the phpLD install. That way it gets the info it needs.

You should know I don't use IM, I'm just a crazy linux guy who doesn't back up his data :P

If anyone has any suggestions or comments let me know.

-Joshua

[f1gm3nt]

Just want to give everyone an update. Right now I've got a bug that I'm working on. Once this bug is fixed it should work and I'll release a new version of it. The bug is when it goes to check on categories and doesn't do it right and it ends up adding the same category a bunch of times.

-Joshua

UPDATE This bug has been fixed. The problem was that it would check the title and not the URL title. This script will replace certain character that it finds in the URLs and replaces it with the correct letter or whatever you want it to be.

I want to test this a little more before the final release of it. If you want a test copy let me know by posting the request for a TEST copy or PM me. It should work but I am testing it to make sure there aren't any more little bugs lurking around to piss me off later :evil:

-Joshua

[jminscoe]

I would like a test copy

[jminscoe]

So far it has crawled and I have about 40000 links so far I haven't converted them yet but this copy seems to be going without a hitch so far will keep you posted

[f1gm3nt]

:shock: that's a lot of links!

I haven't tested it with that many yet, but keep me posted. Once it's finished crawling converting might take some time. I haven't tested it a lot with large links like you're doing, but if the convert.php page times out you should be able to reload without any data loss or any duplicates going in.

Keep me posted either by here or PM me.

-Joshua

[jminscoe]

Ok it converted most of them you can see it here http://www.ahealthylifestyle4u.net/directory/
the following is what it did

Table Records
PLD_CATEGORY 6,817
PLD_LINK 62,176
category 6,910
dmoz 6,911
links 66,120
One more question can I delete category, dmoz, links now or what

[f1gm3nt]

Yea, after you done with everything I would go ahead and delete all the files you uploaded and also the three tables(saves some space).

I am kinda wondering why the row counts are different for the links. I'm assuming that it doesn't import them all :cry: I'll have to look into that.

About how long did all that take?

-Joshua

[jminscoe]

On my pc it took all night to import :( to convert about 30 minutes I just had to refresh a few times I think the difference was because it couldn't connect b/c when I checked it said page not found

[f1gm3nt]

heh, that's awhile! Let me knowif there's anything I could improve on it. In the future I would just like for people to just upload and run it, but for now it doesn't.

-Joshua

[jminscoe]

I am not that savvy on code but considering I was dmozing on Health I think it did a great job and considering I had over 60000 links to crawl and import what can one expect

[f1gm3nt]

perfection? I guess it works, if anyone else wants this thing then PM me and I send it to you.

-Joshua

[Spiderman32]

Minor fix in crawl.php

echo "

Crawl complete! Now run <a href=\"covert.php ---> convert.php

[f1gm3nt]

Yea, I noticed that, it should be convert.php with an N. Long night of working on it. Sorry about that guys and gals.

-Joshua

[oneant]

hi,

first, thank you for sharing your mod to the community.

i tested the mod and it worked great, i then edited the URL of the directory to crawl on the crawl.php and uploaded it on the site and refreshed crawl.php but nothing gets crawled.

what i want to do is append the old database with the new data, is this possible? if yes, how?

thank you

[f1gm3nt]

If you are wanted to append your current phpLD then just upload everything and it should just append your directory. Be sure to back it up first just in case something goes wrong.

If you've read the ReadMe.txt file and edited the two files(class.dmoz.php and class.sql.php) and uploaded them then you should be fine. You of course also have to install the tables that it uses by using phpMyAdmin or some other means.

Once you have finished 'crawling' it will tell you and then you need to run convert.php. This converts all the data and puts it into the phpLD tables. It will check to see if the category is already there and if it is it will not add the category.

Please let me know if this helps or please submit some more detailed information about your problem.

-Joshua

[Jim_Westergren]

I have made this another way with a PHP script I found on hotscript.

http://www.addurl-free.com/odp/

Actaully quite simple.

Here you can download it: http://www.bie.no/products/phpodp