JBidwatcher is a snipe tool but also a monitor tool. It really helped me a lot for BOTH function. In fact, I began to use it back for monitoring only since I use other snipe solution for now (mainly because JBidwatcher failed me in the past because either of the program or my computer that was not reliable, and also because I have multiple accounts… and yes, I have run multiple JBidwatcher with multiple accounts in the past, but the first reason still applied at some point).

The biggest problem I have with it since always, is that it become excessively slow/unusable when you have a lot (I was gonna say too much, but, I do not think even hundred thousands or millions of auctions should be a factor that influence that much its performance, read it should scale, unless we decide to qualify JBidwatcher as a temporary tools and not an archive tool, which can be fine, and maybe that is a good answer, but how hard would it be to keep an archive without affecting the performance? I think not much).

So, I wonder, can we have an “archive auction” option, and make sure these are not actively loaded when JBidwatcher starts. An alternative could be a “search active” option that would let us do just that… it open a windows, you enter your search criteria, it list the auction.

I am quite a perfectionnist when it come to develop software, so I would say, maybe the correct name is not “archive”. We will want to keep active some ended auction (I guess we can qualify of non-archived). An alternative name might be “active”… but usually when you “archive” something, you put it aside. But I might give some deep thoughts later about it.

For now, Morgan, if you can only tell me if I could just go in the database, put a flag, and make sure when it open it does not load these auctions. I will be pragmaticelly happy. Might be considered a hack, but it is not that dirty if we do it “right” even if implementation is not complete. I mean if data are OK, but interface is not full featured to use those, then it is still clean. Later we can decide how we interact with the archive…

I will probably just re-delete my completed auctions, but I really do not like to do this over and over… and merging all my old backups might be an hard task (unless I am in a mad enough state to do it! :D).

So is it possible to do something quick about this?

I somehow posted about it years ago, but never got real feedbacks. I can do it myself too if you give me some hints (or I may do it anyway). If you can load the completed auctions in a second step. You can probably take the decision to NOT load these completed auctions or archive auctions quite easily if you know where the code is (or like I said, if you can tell me hint, I can at least make my own patch fast).

I remember when I looked at the database long time ago, I think you had one table that was for backup auction or something… so I guess you thought of a similar functionalities.

Oh, and here is the post I did about it:
http://forum.jbidwatcher.com/forums/1/topics/3525
http://forum.jbidwatcher.com/forums/7/topics/4675#post_12059

Thanks

Greetings,
Well…yeah, I got an email recently from someone who’s complaining it takes HOURS to launch JBidwatcher. Unfortunately the reason is that their database contains 1.4 million auction entries, and JBidwatcher tries to do some joins on that data early on.

So…man, I could go for hours about the stuff that’s built up in JBidwatcher over the years, and the cruft that’s in the database right now. (For example, it doesn’t delete old [auctions] rows, so they stack up, even though nothing refers to them anymore.) My 3.0 version intends to significantly reduce that, by not creating a new [auctions] row for each update. That’s how it works in the 2.x series, it creates a new row and moves the [entries] foreign key reference to the new row. It used to delete the old one, but something broke and it started deleting active rows, so I turned off deletion until I could fix the deeper issues.

If you use the Derby database, you can do some really straightforward operations directly from Derby’s own interactive console. The derby username and password are ‘user1’. (This isn’t particularly intended to be secure as it requires access to your box to get to anyway, and the username and password are in the clear in the source code.)

Personally I hate all GUI database browsers; they’re all terrible, unintuitive, and crashy. That said, you could use Squirrel SQL to browse and issue SQL commands. Speaking as a Mac user (if you need Windows or Linux help, I can give it), first go to the Drivers, and add derby.jar from JBidwatcher.app’s own directory (Contents/Resources/Java/derby.jar) as an Extra Class Path for Embedded Derby. Then create an Alias, using jdbc:derby:/Users/{username}/Library/Preferences/JBidwatcher/jbdb as the database, and user1 as both username and password. A test should be successful. Hit OK, and Connect.

Switch over to the SQL tab, and type SELECT * FROM auctions; and press Ctrl-Enter. That’ll query your [auctions] table. Similarly SELECT * FROM entries; will query your entries table.

You could do something like:

CREATE TABLE ARCHIVED AS SELECT * FROM auctions WITH NO DATA;
INSERT INTO ARCHIVED SELECT * FROM auctions;
CREATE TABLE ENTRY_ARCHIVED AS SELECT * FROM entries WITH NO DATA;
INSERT INTO ENTRY_ARCHIVED SELECT * FROM entries;

to copy everything in the existing [auctions] and [entries] tables to an [archived] table equivalent which JBidwatcher won’t ever touch (because it doesn’t know about it). Then something like

DELETE FROM auctions WHERE created_at < '2014-12-23 00:00:00.0';
DELETE FROM entries WHERE auction_id NOT IN (select id from auctions);

to prune to the most recent listings. (In that case, ever since the night before the night before Christmas.)

The key understanding here is that an Auction is everything that comes from the server. The information isn’t added by the user, or JBidwatcher, or anything like that, it’s what we glean from eBay’s listing pages. The Entry is everything around it, overridden shipping price, last bid amount, category, snipe info, etc…

I had some ideas for archiving auctions, but they never came to fruition, so it’s not terribly helpful right now. My hope is that with the 3.0 branch I’m working on, I’ll be better able to do a lot of this, especially as I’m moving vast amounts of the database logic down into the scripting layer. We’ll see…

In any case, back up your stuff, seriously, before doing any of that.

It’ll probably be easier in the 3.0 version to do database operations, as it has the whole Scripting Manager interface which is AweSome. :)

— Morgan Schweers, CyberFOX!

Thanks Morgan, I am looking into it. I had some problems today with JBidwatcher simply not loading and freezing at the part that seem to load UI (I updated all my JVM, and only 1.7.x and 1.8.x remains, and I doubt it was using 1.6 before… but I have been bold and deleted, hehe, but I might retry with that later)… I will probably have more questions or even suggestions later, but I will probably give more effort in your new version. Me too, I might have some project related to eBay, and contributing can be a good way to learn.

That being said, quick questions to start… if I understand right, auctions are only the data about an auction, independently from everything I do? (well I see flag as “outbid” in the database, but it mights not mean what I think).

Q1) So, if I do not do the manipulation in the AUCTIONS table… then will it load all these auctions? There is like 262586 rows for now in mine. So if it is doing a full load, that might be a first problem. And is it needed to load all this? (is it just in case I re-load the auction in my entries).

Q2) Is entries corresponding to what I see? I had the first reflex to try to archive/delete only entries that were in my “complete” tabs before a given date… then I started JBidwatcher, but same things (my problem might be elsewhere, but I did not play with anything much, except DB and updated my JVM thought, which might be considered a major change, I do not know how good the Java world is with keeping being backward compatible with old Java version). Of course, if it load all auctions, then that won’t help.

P.S. FYI, I am quite good with db/SQL/modelization/etc. (I am picky about naming, etc. and go a bit beyond most to get it “right”). As for SQL, I will try to do some project with a so called “NoSQL” DBs… projects that might scale a lot of records… but in the end, it should not change our modelization much. I also have done lot of web development, and I do not know if you did a lot (most common software dev jobs touch this nowdays), but when doing it, we are use to load only what we need. The way you talk, you probably know all the reasons it is like that and might know how to solve, but choose to create a new major version that will fix what you want to improve.

Any ETA for 3.x? Or is it already usable and source available?

Also, I saw somewhere you said you were planning or working on another project related to eBay, etc. Have you move forward with that? Is it live? Or still a work in progress / idea?

If some answer, like for projects, are not thing you want to answer publicly, you can always email me for those type of discussion. Of course for what is related to JBidwatcher, unless there is a reason, it is better to keep it public so everyone can benefit. ;-)

P.S. My email is the same as my website “domain” (in my profile)… except that you replace the first dot with and @. Or you can probably fetch it from your DB. ;-)