Saturday, August 28, 2010

New Downloader

Besides working on combat, I played a little more with Jetlang and created a downloader, which works asynchronously and can inform interested listeners over PropertyChangeEvents.
Jetlang's concurrency approach is centered around the actors (who performs a task/where is it performed) and the tasks instead of around the data. Normal multithreading requires you to synchronize on the data you share, while an Actor framework doesn't share data but pass it around, so that only one thread "has" it at a time.
This makes it easier to code, because it is how humans think: You don't worry that two people will write on the same piece of paper at the same time; you wait until the piece of paper is passed to you. That said, it still needs a lot of thinking: Who are the people in your program, what's the piece of paper, and when should you pass it around?

Btw, I made a little comparison between the new downloader and the one used by forge (admittedly, in which I also had a great part).

Program           After 2 minutes  For 1500 images
Laterna Magica    1500 images       2:00 minutes
Forge              200 images      10:00 minutes

I looked at the code for Forge and checked if it is roughly comparable to mine, and it was; otherwise I wouldn't have posted this. I changed two things, however: I increased the buffer size to 8 kB to match Laterna Magica, and I removed a print statement. These are pretty slow and shouldn't be there when measuring time.

Laterna Magica's downloader uses 10 threads, which means that 10 pictures are downloaded at the same time. Even though it might seem at first that it shouldn't make much of a difference, because the total bandwidth stays the same, the numbers clearly say that download speed is five times better. The reason for that is the reduced overhead. Every picture needs the following steps to be done:
  1. Connect to the server
  2. Request the image
  3. Receive the data, store it into a file
The time is lost between step 2 and 3. The first two steps has you upload data (the request), the third has you download data. Between that is only waiting. When using only a single thread, this waiting directly influences the download speed, because nothing else happens in that time. Using multiple threads, waiting only blocks one of the threads, leaving 9 downloads running in the meantime.

Besides downloading card pictures, I adapted Laterna Editor to use the new downloader, which also shown its flexibility: Laterna Editor downloads HTML pages from Gatherer, but doesn't save them to files. Instead, it further processes them to compact card descriptions for Laterna Magica. While it took a little work, the downloader is not limited to Files but can download to any location, such as a string buffer in memory.

The card downloader from Laterna Editor also shows how the task-centric approach helps with programming. This picture pretty much describes how the download process works:


Both Source and Destination are classes implemented by CardDownloader, but implement interfaces specified by Downloader. The source checks if the URL is a search or a single card, and also handles the automatic redirect if Gatherer only finds a single card for a search. The destination saves the HTML page to a string and waits for the download to finish, by registering a property change listener to the download job. If the download was successful, it sends the page to the search- or card processor, depending on the source. If it wasn't, it just discards the downloaded page.
The search processor finds additional result pages (every page has at most 100 cards in the compact view, which is used by the downloader) and of course the individual cards. The download of those is again done by the same process. The search processor simply posts the DownloadJob to the downloader and that's it.
The card processor finds the individual fields like name and mana cost, and stores them into a card. Once that's done, it posts the card to a Channel where the Gui can find it and in turn show the editor.

1 comment:

nantuko84 said...

very interesting. actually magicwars uses the code from forge (I took it long ago as I needed setting proxy). What I still don't like in forge version is that it's modal dialog (in my version is was non modal, but didn't had proxy), so you can't play the game while card pictures are being downloaded. it is annoying when you need to download 2500 card pics...
What I changed in forge version was adding average size of download in download progress bar description (smth like "Downloaded 3 of 2500 (~250 Mb left)") and, that is very user friendly, the option to download cards for current game only.
but your idea about several threads is very good, as it allows to use network all the time.