lano1106 OF V2 5.75.5G journal

[lano1106]

Cycle 20 (v2 cycle #2), day Off:

It has been the bug fix festival today. I did fix a lot of small glitches that the last run did trigger. It is amazing that I have been fixing bugs for months and there still new one that are showing up. I guess this is expected with the level of complexity that my system has reached. Over 75,000 lines of code.

But overall, I'm quite satisfied of the quality and performance that it is now at.

[lano1106]

Cycle 21 (v2 cycle #3), day 1:

I did add another small functionality to my server and this did also allow me to discover another small bug.

Another detail worth reporting, it is that I am now at $60K 30 days trading volume... It took me over a year to reach the first $50K milestone and it seems like I could reach the next one at $100K in the next month.

The way that I see it, it is only a matter of cementing even more my confidence in my trend trading skills and as soon, as I feel ready, simply double the amount that I trade and this will bring me to $100K volume in no time.

Something that I feel is that since OFv2 started, I may trade more rationally and with less emotions which is good thing.

[Shannon]

Congrats, man! I'm enjoying seeing you succeed with this.

[lano1106]

Cycle 21 (v2 cycle #3), day 4:

The last 2 days have been super productive.

I bought some Monero coins that I wanted to transfer in my ledger wallet. I did discover that the ledger client does not support Monero, you need to install the official client and ask it to create a wallet from the HW key.

I did use this opportunity to do many tasks related to Monero. That is setting up monero miners on every machines running idle in the house.

Everything that I did try was a success.

I'm running a monero node on a server to which I connect the monero-gui client through a ssh tunnel.

I did create 5 xmr mining workers but it felt a bit inefficient to make them all connect on the pool. I did therefore setup a mining proxy which connects to the pool on the behalf of my 5 workers.

Everything that I did setup is working like a charm and I can now set a cross beside this task that I have wanted to do for some time...

[lano1106]

Cycle 21 (v2 cycle #3), day 5:

Still fixing bugs in the code. I had a log file waiting to be reviewed since Thursday.

There was a bunch of glitches during the execution. It is not that the quality sucks. It is simply that it is a very complex problem and the exchange server is continually giving me behavior that weren't expected and never seen before.

Beside that, I got a bunch of new improvements ideas that did flow out of nowhere in my mind this morning. Very excited about them. The problem is that the ideas are coming faster than I can execute them...

[lano1106]

Cycle 21 (v2 cycle #3), day 1 Off:

Setuping my XMR miners did make me realise that crypto mining was something fun that I did like.

Something that I did want to do for a long time but I have never dare doing it before due to the price tag was to purchase BTC miners. I just bought 2 Antminer S9...

There was definitely a fear that investing in that was a resource misallocation... I did just put the fear aside and I did pass my order impulsively.

I started to look around if I could place them behind a proxy like I did for the XMR miners. There is no such thing for the mining Stratum protocol...

I did look into the stratum protocol for some time yesterday and I have found a very clever way to implement a proxy for this protocol.

When the proxy receive subscription reply from a mining pool, You slightly modify the answer that you are going to send to your miners.

1. You decrement the Extranonce2_size value
2. You append unique 1 byte id for each miner to the Extranonce1 field.

That should do it. In fact, if I put that idea on github, I suspect that this could become a very popular project...

But I'm not forgetting my trading server project. I did finish reviewing the logs. This time, I have found a bug in the exchange code. I did report the issue, lets see what they are going to reply. This could deserve me a small bug bounty. They did that last time that I found something similar.

I have restarted the server before going to bed. Now, I see a lot of opportunities that I fail to grab. When I initiate my trade, the spotted book entry disappear 40-50ms after having appeared and I miss it. It feels like someone else is spotting the same opportunities and I am getting front-runned.

Timing is perfect, I started to have plenty of good ideas for optimization. It seems like it is going to be optimization season now (one of my favorite hobby to not say one of my sins...).

First step has been done. I did put in place some instrumentation code to measure the time it takes between receiving an update from the exchange and the time an order is sent out.

Once I know the answer, this is going to be my baseline to measure potential improvements (or simply if there is a lot of fat there. If there is not much, it is simply a question of getting my server closer to the exchange).

But things got crazy since I restarted my server. Usually, the log file size gets around 5MB after a week of running... Traders are crazy since yesterday evening because the markets are taking a dive. The side effect is that the trade volume is so big that it is saturating the exchange server and as a sideeffect, it sends out of date data. My server went crazy because of that and it generated 17MB of logs. No idea what did happen exactly. I'll need to look into this damn log file. I was so happy to have finished goign through the logs of January 7th.... I'm not too happy to have to return to log analysis so fast after having done it for a long time.

Fortunately, by inspecting my account balances, nothing dramatic did happen. I made even have made a small profit but I need to find out what did happen. The server did shuffle my balances around the place... It is weird...

Oh well, my days are everything but boring.

Update:

I have a first reaction time number. It seems a bit high... It took 2ms between receiving and update and sending out an order... It is going to be fun shaving this number...

[lano1106]

Cycle 22 (v2 cycle #4), day 1:

Another big day today and yesterday.

I have tried to change my threads scheduling from regular to real-time to the give my server VIP access to the CPU. I have found out that it is not possible because my server is in a virtualization container.

I am considering upgrading my hosting plan from a VM to a dedicated server but before taking this decision, I need to have more insight about by how much time I am frontrunned. My reaction time is about 2ms. With a dedicated server I'm confident that I can reach a time below 1ms... but if I am frontrunned by 5ms+, my issue isn't software. It is latency and I'll need move my server closer to the exchange to fix that. Honestly if this is what needs to be done, I don't know at what door I can knock at to receive help but if it is what needs to be done, so be it...

New code, with new instrumentation to get the answer that I am looking for is baked in and is running since this afternoon.

Right now, my main issue is that the server is trading so much that I am getting overwhelmed with logs to review... I guess we could call that a high quality problem...

However not all the latest trades were good. With all the recent changes, I did introduce some regression and this has caused some bad trades. Fortunately, I have found what it is and I have patched the problems.

I think my 30 days trading volume is now close to $86K... I'm very close to the next lower fee level at $100K. I expect this to flood my system with even more trades... 1 year to get to $50K volume. 1 Month to double it... That is a prime example of the snowball effect...

[lano1106]

Cycle 22 (v2 cycle #4), day 2:

I started to dig my monster 17MB log file. I start make sense out of it. Basically, there has been a bug in the exchange where it seems it was sending ETH/USD price to the LTC/USD pair.

Well, if LTC was suddenly worth $1000 instead of $150, you can bet that my system was crazy...

Hopefully for me, a small glitch did stop the system to go all in in this fake signal.

Very rare event but I devised a design for a protection. On top of avoiding falling for those fake good deals, also very importantly, it will prevent to fill the log

[lano1106]

My new protection against bad market data is completed. 13 source code file updated and I created a pair of files to include the new created class.

For few days, I have noticed that I'm becoming fearless in my work and as a result my productivity has increased.

Engineering the type of system like the one that I am developing is very complex, last time that I did check, I was at up to 75,000 lines of code. Every small change can introduce a bug that can be costly. I feel like this thought can induce hesitation to make changes.

Right now, I feel like I totally got rid of that fear. It is really fun to code fast and it is rewarding to try out new features the same day that you did think about them.

It must be something that I must attribute to OFv2.

Update:

I have collected the data allowing me figure out if switching my server from a VM to a dedicated server would make a difference. The answer is no.

My server response time is about 2ms and the time it takes between an entry appearance on the book and the exchange accepting my order is 140ms. My fastest competitor appears to be able to snatch those entries in about 40ms.

So, I have a 100ms deficit. My most urging issue is therefore network latency. I'm going to search for a solution...

Also, I have discovered a very trival, obvious but critical bug that was stopping my system from seeing trade opportunities. It was in your face type of bug that must have been in front of me many times but I was blind to it.

I guess, this could qualify as subconscious sabotage... It appears that OFv2 is able to eliminate those and that I am getting very close from experiencing major success...

[lano1106]

Cycle 22 (v2 cycle #4), day 3:

Many things did happen all at the same time.

I am really close to break the second transaction fee level by reaching the $100K last 30 days trading volume. I am currently at $87K.

It is crazy how some glitches that are in plain sight just go unnoticed because of the assumptions that you make. You just add some log to track something totally unrelated and it makes reevaluate your assumptions and then suddenly you get to see the obvious and what is wrong with the code is doing.

It did happen just right now.

I like the process. This is how you learn all the nuance of a certain field and become an expert.

As I was looking for ways to measure my latency with the exchange. I started calculate it locally. The problem is that it makes me query the system time through a costly system call. This is a bad design since I process millions of updates every few minutes. That means that I am doing millions of system calls every minute.

Then, I tried to figure out if I could not instead use the timestamps contained in the updates. By evaluating this idea, It made "see" that the condition to reevaluate the trading opportunities affected by this update was flawed.

Next trade server upgrade with those changes might affect the trading frequency of my system.

[lano1106]

Cycle 23 (v2 cycle #5), day 3:

Wow, I took what I think to be the longest journaling break since I started OF.

I feel like I am in a weird pass. With the lockdown measures intensifying with the curfew which doesn't make any sense on a public health level, I feel like I am more disconnected from reality or my expectations of what reality should be. All the rules around me appear to crumble.

I'm not afraid of that... Sometimes breaking rules open up new opportunities.

but it is weird... I'm experiencing giant progress in my trading project. The last issue that I was concerned with is mostly addressed now. A lot of things unblock but some others are stalling and time seems to pass super fast but at the same time, by being under a strict lockdown, I kinda have lost most of my references about time...

I feel like in a dream... Not ideal situation but I guess it could be worse so I won't complain...

Update:

I spent a whole day writing a small dialog window... I hate wasting precious time on something like that...

[lano1106]

Cycle 23 (v2 cycle #5), day 5:

$100K 30 days trading volume milestone has been reached yesterday. The next goal to reach is $250K trading volume in 30 days... I wonder how fast this is going to be done...

My pro account application has been approved. My new daily withdrawal/deposit limit is now at a ridiculously high $10M. Who knows maybe someday I am going to hit those caps.

Those new lowered transaction fee are making very juicy arbitrage profit. The last 10 trades have all been profitable...

My code had a 'fast exit' feature that would make the system exit a trade when it finds that it is not profitable anymore. It could transform a painful 5+% loss into something mild like < 0.5%. It wasn't working at all last week and I had to endure several 5+% loss to figure out what was the problem.

It turns out that it was a problem due to a crunchy multi-thread race condition that was exacerbated by the fact that I upgrade the server I am running on was more powerful than the previous one.

Oh, and I rented a VM in the Sillicon Valley area. This does improve a lot my reaction time. The race for a book entry has now become fair and I win a good amount of time this race. I'm on the verge of upgrading from a VM to a real physical dedicated server... The only verification that I need to do before doing it is checking the latency I get from running on AWS.

Based on all these good news, it seems that I'm living a princess dream that comes true... Not exactly, I have few serious concerns but I decide to focus on the positive... This is helps keeping my happiness and my satisfaction in lifre.

[lano1106]

Cycle 23 (v2 cycle #5), day 6:

Very quiet day in terms of trading... Exchange volume was relatively low at around $820M traded daily. I say relative because I remember back last October when I was blaming the low volume around 200-300M.

Things are relative. For about a month, trading volume in the range of 1.5B-2.5B daily has become the norm....

I fixed a couple of bugs today but I have stumbled into one that makes me speechless... I have several execution strategies. A strategy is choosen depending on current market conditions. The one the system use when the conditions are very favorable is broken. You cannot tell that something is wrong by looking at the logs but being in front of the screen as the execution is going on, it is obvious that something is broken...

idk since when it is like that. In some way, it is a luck that I did witness it. Now that I know that something is wrong and that I know what to look for in the logs, it becomes crystal clear something is broken. However, I did look at the code the whole day and I cannot tell what is wrong. I have noticed something that hints about what could be wrong. I have my ARB trades and my more casual trades. The difference between those 2 types is that for the ARB trades, I do set a ceiling price to marks the break even price. With the casual trade, I leave this field blank as I just want to convert currency A to B without any performance goal. The execution strategy works fine for the casual trades but not when the break even price is set....

Usually, I am very stubborn and I don't let go until I find what the problem is but for this one, it didn't work. I enabled some extra logs in that code area. I just need to have the code to get executed during the night and I will have new insight into the problem...

Bottomline, something is broken in the execution and I don't know since when it is that way... That could explain why I got some bad results for some time...

I know that I will benefit when the problem is going to be found but in the meantime the effect of this unexpected investigation is that I have postponed a lot of others tasks...

[lano1106]

Cycle 23 (v2 cycle #5), day 1 Off:

I have found my major bug that went unnoticed for months. This one major discovery and I wouldn't have found it by looking at the code.

It is some weird behavior about how double arithmetic works.