Unbelieveable

At work we use third party content for the products we sell. The match this content to the products, this third party, Icecat, uses XML index files. Our current setup is that a VBScript inserts the complete indexfile (~150MB) into our MS SQL database. Now everytime we put a new product in the CMS and put it online in our webshop (its my job as a content manager, I don't own the company), it queries for the location of the XML file with specs, so the CMS can use this XML to output specs with the product.

So you see, it's quiet vital to have this import working. However, since last Wednesday the import stopped working. According to the IT guys, there was no error, it just stopped. The company who made this specific VBScript and the CMS we use said they would look into it. Today, a week later, they finally got time to do this. What they found is that the indexfile got to big, and then they informed whether there was a daily or weekly indexfile (=smaller and therefore should work).

OMFG! First of all, there is a daily file, so they should have implemented a daily incremental import in the first place. Second, VBScript's default XML parser is DOM based. This means that the XML document will be extracted as a tree-structure into the memory, and for every read or write action, the amount of memory required increases. So before the import quit working, it used to take up to 10 hours to finish.

Look, I don't have a degree in programming, I didn't do any education remotely close to this. I am, by al means, a n00b when it comes to scripting and programming. But if I know all this, why the h*ll can several people who finished college in this field, who have their own company and ask a lot of money to even answer an e-mail to us not figure this out?

An argument given was that it would be too much work to rebuild the script (75 lines long) to use a SAX-kind of solution (which AFAIK isn't available in VBScript). The server in question also supports Asp.net 1.1. So XmlReader is available. Although not the same as SAX (pull instead of push), it is definitely comparable in performance, or faster. In fact, I use this solution to read the same indexfile and match it to our outstanding requests with Icecat in under 7 minutes.

* qrazi is about to get so frustrated for being underpaid, having a job a monkey could do, and the meeting people who pull this crap, but make three times what he makes, that he's thinking seriously about just plain quitting this job....