Attachment repair

[Tom McIntyre]

I managed to come up with a technique that would replace one red x image in each post that had them. With that it was simple enough to run it repeatedly until all of the red x's it could find were gone.

The images currently appear at full size in the posts, but they will be converted to thumbnails when all have been found. That job is done by a currently existing program we have available on the site. If a post with the large images is edited and saved, all the images are converted to thumbnails immediately.

In a brief check, it seems to have missed a few and I repaired some of those by hand to make sure that the content was actually available.

This process repaired about 1,800 posts and probably 9,000 missing attachments.

There is more than one type of error and each of them left a different fingerprint. In particular there is no trace of the missing images in some posts that had them, but I do have the list of those and which attachments were there. In those posts, the attachments will all appear at the bottom when restored, which is probably where they were originally.

If you see posts with red x's please post a link to them here, so I can try to figure out what their story is.

I am working on the images first because they have the most impact on viewers. I will also be able to use the same approach to repair the references in some threads to other threads and posts on the message board.

Please do not post about those until I can get set up to work on them.

 

[Jerry Treiman]

Thank you, Tom.

The following threads still have red X’s:

Waltham's Riverside grade -- the 12-size variants
Some of the posts look like they still have the correct thumbnails but post #29 has no thumbnails.

Waltham 16-size split 3/4-plate
red X in the first post. (I still have the image if it needs to be replaced).

I will keep exploring for others.

 

[Tom McIntyre]

I should have the images in my file store. If it is not there, I will let you know.

 

[Tom McIntyre]

Jerry, on your reference and quite a few others (a bit more than 12,000), I had overlooked that the new site requires a URL to be secure with https instead of http. If the request arrives on our site from the world, it usually gets converted on the fly, but it was not working for the images that were previously imported on the old site. Those should all be displaying as large images now and will become thumbnails when we do the final pass.

 

[harold bain]

Jim, until we work out a solution to the picture problems, your warning is fine. We probably should have something similar on this site. I suspect our problems are rooted in switching from http to https, a more secure format. But I have faith that our techies will find a solution.

You mean I was right,Tom?

 

[Tom McIntyre]

Harold, the group I just fixed were not working here after I put the material in the right location because I forgot to make them https. I think the lack of conversion was related to some of our images being imported with the Import External Images tool that we used to capture off site references.

The other biggest group are the ones that were incorrectly inserted on the old site with an incorrect but functional attachment syntax. I still have not cleared up all the fallout from those errors, but this time I will be more careful with the https.

We will still have some missing images that were posted during the time we had Insert External Images disabled because it had been unreliable with several of the hosting sites. Some of those where the images still exist on the hosting site should be recoverable. As an example I found an image on rrstd's web site that had been hosted on his Charter.net page when it was originally posted. For the watches, I may be able to do that for some of them. I am not likely to be able to find many missing clock images that are out there in netland.

 

[John Matthews]

Tom,

I am a little confused as to the state of the repair.

I like to keep things simple.

If I look at 'My Attachments' which is where I check to evidence of progress, I find that all of my attachments appear to be present up to those I posted on 30 March 2016, I then I have a gap and the next date stamp on the next attachment is 29 September. So, as far as I can tell all my attachments between those dates are not attached to their posts and none this far have been located and re-attached.

Please confirm that I will be able to see that my attachments have been 'recovered' when they appear in 'My Attachments' and I don't have to check the individual postings.

Regard,

John

 

[Tom McIntyre]

John, I believe your images are all in place. I am waiting to convert all the image references to attachments as the final step in this process. If you could check some of your early PDF containing posts, that would save me a bit of time verifying the PDF behavior.

 

[novicetimekeeper]

fantastic job, well done. My favourite clock is fully restored.

 

[John Matthews]

Hi Tom,

As requested -

I first checked the list in 'My Attachments' no pdf files are listed in the gap period March to September. To be sure I also checked some threads ...

English movement & frame makers -> pdf files missing 10/9 and 11/9
Non-Standard English Lever Escapements from the first half Nineteenth Century -> pdf file missing from 21/4
Roskells of Liverpool and London -> pdf files missing 1/8, 3/8 & 5/8

I assumed you wanted me to do it immediately ... if you did I cannot see any evidence of the repair,

John

 

[Dave Coatsworth]

Tom,
I'm a little confused on what was repaired. If you look at my favorite Illinois Private Label thread: Illinois private-labels, most images are still either missing or have an "View Attachment" tag which does not work. I really don't see any improvement.

 

[Tom McIntyre]

John, there may be a separate issue with pdf files. The system does not handle them the same as image files unless they were inserted with the new software that asks if you want an embedded image. There are only 22 such files in the database and they are all yours.

It will be easier to just do a small edit to each of those posts than to figure out an automatic procedure. The people (in Germany) who wrote the new pdf attachment handling are continuing to work on it and a future update may take care of the issue of listing in the attachment list.

 

[Dave Coatsworth]

Interestingly, the Hamilton Private Label thread is much improved. (Hamilton Private Labels, Please add to the list). I did see some red X's on posts 127, 145, 170 and 171.

 

[Tom McIntyre]

Dave, I am a bit confused by what happened with the Illinois thread. I had been using it as my test bed since it was a very long duration and covered a number of ways of handling attachments.

We have some programs that run nightly and "clean up" bad data. I suspect that one of those hit that thread. To reassure you, there is nothing lost. I still have the original from the old site with 1,028 attachments in it. Whatever happened there probably happened elsewhere also. I will come up with something to put them all back. (I thought I was nearly done. )

 

[John Matthews]

may take care of the issue of listing in the attachment list.

I'm sorry but I don't understand this issue - the pdf I attached here on 28/10 is correctly shown in my attachments list ->



so I don't understand what issue they are hoping to fix. I haven't had any attachments problems with the new software - in fact it's a piece of cake!

What do you mean by an edit - do you mean locate the individual posts and re-attach a new copy of the file. As you have identified the 22 examples that's a good start. Can you retrieve copies of the original files? as some may be early versions of documents that I no longer have. If you don't have or cannot locate the files, if you send me a list I can see what I still have. I don't have the permission to do the edits, so I assume you or one of your helpers will have to repair the posts.

John

 

[Steven Thornberry]

nearly done

How do you define nearly done?

 

[Tom McIntyre]

How do you define nearly done?

For about an hour just before I posted that everything was working. This software has some automatic processes that run in the background from time to time. I think one of those clobbered the fixes I made.

It feels a bit like two armies fighting over the fairgrounds where all the school kids have come to play.

 

[Tom McIntyre]

I believe we are making progress on recovering attachments. For the past week many of the attachments have been replaced by the large images of the material. We are now running a process in the background that is changing all of those into attachments at the location where they appear in the posts.

The same process is trying to process all IMG insertions but in many cases those are from PhotoBucket or Flickr (and other hosting sites) that either no longer have them or no longer allow them to be served publicly. In any case there were just under 28,000 posts with such material and it is all being processed at about 240 posts each hour. That process should be finished in a few days. It is looking like there may be over 2,000 images that the hosting sites refuse to display. It is possible that some of those will be recoverable with the help of the account owners.

There are another 10 or 12 thousand posts that had images before that were not inserted in the text. When the current process is finished, those posts will have their images replaced at the bottom of the text and then the same attachment processor will be run again to convert those to attachments.

The final step in the recovery will be to recover all the attached PDF files. There are not nearly as many of those.

for those who had images in the gallery on the old site, that material is still safe, but I have not yet determined the best way to recover it.

I will update this thread as we reach milestones in this process that you can observe by reviewing threads.

 

[shutterbug]

Thanks for all your hard work, Tom!

 

[Tom McIntyre]

The initial pass that converted all discernable bad image references has completed. Where it was not possible to recover an image, it was almost always due to the image source no longer being available. Most of those were older references to images on Photo Bucket and some other image hosting sites. A number were also references to my old awco.org site which has been out of order for a while now.

Where these exist there may still be red X symbols for a missing image. I would appreciate a lot more eyes looking for the ones you think are important. I do have the information from the process that nominally lets me find them. i.e. I can see where attempts were made to correct Photobucket references. What I cannot see is whether they were successful or not. The tool reports that there are still 9,441 of those. They should almost all be before 2012.

There are still the images to be processed that were posted as attachments but lost in the transfer. I am ready to repair those, but I would like to clean up the remainder of the visible bad references now.

The invisible missing references and the missing PDF files will be handled in a day or two once the cleanup is done on this pass.

 

[Tom McIntyre]

Update... 2,028 of the missing images have the address of the message board as their content. Those occurred when we first started importing images on the previous board when old dead source sites were found. I will go ahead and replace all of those to reduce the size of the task. I will insert the phrase "Old Lost Image" so those who try to read the old posts will at least know what happened.

This is actually an historic problem from 2012 having nothing to do with the current migration effort.

 

[Jerry Treiman]

Was this the same situation we had where some image links no longer worked but we still had an archive of all of the old images? I recall, a number of years ago, a plea went out for users to identify their images in that vast unsorted archive and identify where they go. I used to have a link to that archive, but I think the last time I tried it, it no longer worked.

I just found the old link - it may help locate lost images on old hard drives?
*http://www.nawcc-mb.com/OMBpics/index1.html*
(I have added an asterisk at front and back to try to prevent it from being a live link - but that did not work)

 

[Tom McIntyre]

It is the same information Jerry. Your link may work if Kevin can get the old machine running again. The BOD is in town, so the staff is probably a little overloaded at the moment.

 

[Jerry Treiman]

I am not sure if you are tackling the following image issues yet:

There is a dead link in this post -
Waltham Colonial A
Will it, and similar links, be corrected in some automated process or do they need to be manually edited?

The replacement for that dead link would be - 12-size Waltham evolution (briefly)
Unfortunately I find that most of the initial images and links in that thread are also dead. Many are of the format “View attachment 123456”. Are those among the ones we still expect to recover or should I try to update this thread with good images and links?

Almost every image, even the links, are gone from this thread -
Enameled American Watch Cases
Do we know what happened?

A number of images are still missing in this thread -
The Riverside - Waltham's most popular movement grade
How can I help replace/restore them?

 

[Tom McIntyre]

The posts where the attachments are missing will be restored on the next mass fix. Those are the ones that were not inserted in the posts, but left at the bottom as we were recommending at the time.

I have pretty much everything in place to do that update, but creating all those new attachments will complicate going back after these broken ones, so I would like to have a clear understanding of them before I do that.

The posts by Schussel in that thread do have broken attachments that show as links, but resolve to an error because those id codes are no longer available. I may be able to figure out how to sdo them also, or I may just do them by hand if the old images are still here (as they may be).

------- later ----------------
Jerry if you check the thread again, you will see that Schussel's images are there. I just corrected those by hand with the same technique that will be used for the missing ones.

It is hard to tell them apart when you open the text for editing. It is only the difference in the index values and the fact that they did not display properly. Of course the post and attachment addresses are all saved in the database that I built to repair them.

I have a database of the ones that are missing. When I replace one by hand, the new software fixes it as an attachment as soon as I click on file.

When I do that with a query for the missing ones they will be left as the URL reference to the correct image and I then need to execute the batch image to attachment converter to make them the standard display.

 

[novicetimekeeper]

I didn't understand a word of that but I'm impressed with your perseverance. Thank you.

 

[Jerry Treiman]

Thanks, Tom. It is nice to see the progress. I can't imagine the complexity of this.

 

[Tom McIntyre]

Jerry, several of the links you mention above are to the nawcc-info web site which was one of the casualties of the equipment failrue in my basement last year. That information is also safe, but it currently does not have a web site to display on because the tools to display the web site as it exists were on the now dead server.

The image links that show up attachment 'attachid' are essentially the same as the ones missing completely.. They were just inserted in the post by the author. I am pretty sure they are all in the repair database also.

 

[John Matthews]

Hi Tom,

English movement & frame makers -> pdf files missing 10/9 and 11/9
Non-Standard English Lever Escapements from the first half Nineteenth Century -> pdf file missing from 21/4
Roskells of Liverpool and London -> pdf files missing 1/8, 3/8 & 5/8

John

Tom - can you provide an update on the retrieval of the missing attachments.

These pdf's are still not present and photographs that I have highlighted that were posted after the Spring software update, such as here Inverted Double Rollers are yet to be recovered.

John

 

[Tom McIntyre]

I had hoped to finish the image portion last week, but the later posts with images use a complex form of reference that requires more knowledge of pattern matching than I have. Another admin @bruce linde has been helping with that and it should be ready soon. Once all the old attachment formats are captured the actual conversion of the 3,000 or so remaining posts will be relatively quick. Bruce has also been cleaning up some very old text attributes and quotation features that have been lying around for about 10 years to make sure we understand how all the mass edits we are planning will work.

The software that manages the PDF thumbnails is a little buggy so I am trying to be careful with it and make sure that we do not make our site vulnerable to attack by using it carelessly. That will take a little longer.

The lists of your attachments that appear in your profile may not be updated correctly as attachments are recovered. Using that list as a measure of progress is misleading. When all the attachments have been recovered, we will try to rebuild those indices but we have not run that process, so it is untested at this time.

 

[John Matthews]

Thanks Tom - I have noticed a few additions to my attachments list but on the basis of your previous comment I have been checking actual posts.

John

 

[bruce linde]

hope to make progress on my portion this week... thx

 

[John Matthews]

I can see that there has been progress and a number of images & pdf have been recovered successfully. It some cases it appears that this has been done manually as there is an edit comment on individual posts. In some, such as here, there is a marker {IMG} but the pdf (in this case) has not been recovered.

Does this mean that there has been an automated process, that has left these markers?

Have we yet reached the stage when you need us to highlight these {IMG} markers so that the process can be completed?

An update would be appreciated.

NOTE - Bracket type changed from [] to {} to enclose 'IMG'

John

 

[bruce linde]

I can see that there has been progress and a number of images & pdf have been recovered successfully. It some cases it appears that this has been done manually as there is an edit comment on individual posts. In some, such as here, there is a marker {IMG} but the pdf (in this case) has not been recovered.
Does this mean that there has been an automated process, that has left these markers?
Have we yet reached the stage when you need us to highlight these {IMG} markers so that the process can be completed?
An update would be appreciated.
NOTE - Bracket type changed from [] to {} to enclose 'IMG'
John

john -

{IMG} is not correct. is.

i went to the thread you referenced about ('such as [URL='https://mb.nawcc.org/threads/roskells-of-liverpool-and-london.144859/']here[/URL]') but don't know what i'm looking for... everything seems to be there.

can you please point me to where the image tags are wrong, or any missing items?

thx,
b

 

[Tom McIntyre]

Bruce, I have been working on this thread this morning. Most of the bad images represent PDF attachment thumbnails that my correction process did not manage. I think there are not enough PDF files for an automated process.

If you:

1. open one of those posts for editing and then copy the URL portion of the IMG tag,
2. you can use the VIA URL attachment button to "upload" that PDF file and
3. (You should edit the "name" of the file in the tool dialogue before "uploading")
4. insert its thumbnail into the post.
5. You can then delete the IMG reference.

There may be as many of 200 of these instances and any moderator or admin who is comfortable following the steps above can fix them by hand.

Users can help by reporting the instances with the correct name of the PDF file. Moderators do not have access to that information and it is extra work to recover it directly from the back end database.

 

[Tom McIntyre]

The attachment recovery project is nearing completion. There are still about 5,500 posts (out of 1.157,000) that have attachments or attachment residue that need to be processed and most of those have either PDF attachments or other file types like spread sheets or word document. There are also a number of odd image files like Windows bitmap BMP files and GIF files that the conversion tool does not handle, so they need to be done by hand. I can do a few hundred of those an hour.

There are currently a total of 1,157,826 posts (when I started entering this one). Of those 78,390 have attachments in them. There are a total of 320,000 attachments on this message board. 32,950 of those are duplicated and appear more than once (as in when someone quotes a contribution containing attachments.) The attachments currently stored use 60.4 GBytes of storage which includes 7.4 GBytes of duplicates.

I have been considering removing all the duplicates from storage. Each of the attachments has a unique fingerprint, so they can be identified independently of any information about where they appear. I am not sure that effort is worthwhile for about a 10% reduction in storage requirements. I only mention this because I thought some of you might find the statistics interesting.

The next project in line is the recovery of the Albums from the vBulletin site and after that the ability to search your attachments by name, date and description to re-use them in new posts. At that point everything here should be at least as useful as the tools on the vBulletin site.

Note that the IMAGE files in the Encyclopedia are a separate structure from the post attachments. Those images are inherently formattable and reusable but we still have not gotten them fully integrated with XenForo. The folks at Vault Wiki are continuing to work on that.

 

[bruce linde]

tom -

thank you for your herculean efforts... i don't think most people realize just how complicated a message board... and transitioning to a new MB... can be. we are extremely lucky to have someone with your skills willing to put in so much time and effort. you set the bar.

what can i... or any of us... do to help with this particular project?

bruce

The attachment recovery project is nearing completion. There are still about 5,500 posts (out of 1.157,000) that have attachments or attachment residue that need to be processed and most of those have either PDF attachments or other file types like spread sheets or word document. There are also a number of odd image files like Windows bitmap BMP files and GIF files that the conversion tool does not handle, so they need to be done by hand. I can do a few hundred of those an hour.
There are currently a total of 1,157,826 posts (when I started entering this one). Of those 78,390 have attachments in them. There are a total of 320,000 attachments on this message board. 32,950 of those are duplicated and appear more than once (as in when someone quotes a contribution containing attachments.) The attachments currently stored use 60,4 GBytes of storage which includes 7.4 GBytes of duplicates.
I have been considering removing all the duplicates from storage. Each of the attachments has a unique fingerprint, so they can be identified independently of any information about where they appear. I am not sure that effort is worthwhile for about a 10% reduction in storage requirements. I only mention this because I thought some of you might find the statistics interesting.
The next project in line is the recovery of the Albums from the vBulletin site and after that the ability to search your attachments by name, date and description to re-use them in new posts. At that point everything here should be at least as useful as the tools on the vBulletin site.
Note that the IMAGE files in the Encyclopedia are a separate structure from the post attachments. Those images are inherently formattable and reusable but we still have not gotten them fully integrated with XenForo. The folks at Vault Wiki are continuing to work on that.

 

[Tom McIntyre]

The main thing that can be done to help is to spot the problems. Some of the older posts have attachments in them that do not display as thumbnails but rather as a link. In many (most, all?) cases these links display an unrelated object or generate an error message when clicked. When there is enough history information available in the database, the original images can be recovered.

If users would use the reporting tool (link in the bar at the bottom of the post) to call those to our attention, it would be appreciated. I may eventually figure out how to recognize them but currently they are just annoying. There are lots of them in the clock forums where there are threads lauding a particular clock maker such as Waterbury. I actually have repaired most of Waterbury, but I am sure there are others like that.

 

[bruce linde]

how about if we create sticky threads in each forum called 'IF YOU NOTICE MISSING ATTACHMENTS PLS LET US KNOW!', and the text inside could have a screenshot of the reporting functionality and instructions?