Chunithm - GetUserMusicApi doesn't always return all results when more than 300 songs have scores #91

New Issue

Kumubou · 2024-01-03T07:46:25Z

Kumubou commented

2024-01-03 07:46:25 +00:00

Recently noticed that scores saved on my profile for Chunithm Sun Plus on the current development build of Artemis suddenly started disappearing in-game. Looking into the issue further, the song scores were saved in chuni_score_playlog and chuni_score_best and did not look malformed, but for some songs the scores would go missing after logging out and then carding back in. Figuring I might have blown up my profile messing around, I created another one to see if there was something wrong with the game, and scores there saved fine. So I then reassigned all of the score entries in those two tables to that new user... and then the same scores disappeared. Weird. The scores are still in Artemis (and even update properly in the playlog and best tables!) but were not showing up in-game. Poking at the scores, I found that they all did load if I had cut down the list in chuni_score_best to a smaller number, and then at some point a bunch of scores would just disappear, all at once. Enabled debug logging in the Chunithm module to try to find out where exactly this was going wrong, starting with the smaller subset of scores, basically going in order from 0 to X in the database plus adding a few scores from later IDs that I noticed would go missing:

203 songs with one score, 25 with two scores, 1 song with three scores

[2024-01-02 21:57:35,064] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1
[2024-01-02 21:57:35,065] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'}
[2024-01-02 21:57:35,089] Chunithm | DEBUG | Response {'userId': '1', 'length': 228, 'nextIndex': -1, 'userMusicList': [snip]

This loaded fine.

235 songs with one score, 44 with two scores, 1 song with three scores

[2024-01-02 22:05:21,122] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1
[2024-01-02 22:05:21,123] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'}
[2024-01-02 22:05:21,157] Chunithm | DEBUG | Response {'userId': '1', 'length': 279, 'nextIndex': -1, 'userMusicList': [snip]

This also loaded all of the scores.

259 songs with one score, 48 with two scores, 2 songs with three scores

[2024-01-02 22:09:53,080] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1
[2024-01-02 22:09:53,081] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'}
[2024-01-02 22:09:53,119] Chunithm | DEBUG | Response {'userId': '1', 'length': 300, 'nextIndex': 300, 'userMusicList': [snip]
[2024-01-02 22:09:53,206] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1
[2024-01-02 22:09:53,207] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '300', 'maxCount': '300'}
[2024-01-02 22:09:53,215] Chunithm | DEBUG | Response {'userId': '1', 'length': 8, 'nextIndex': -1, 'userMusicList': [{'length': 1, 'userMusicDetailList': [snip]

This is when scores went missing. It turned out from my testing that the scores would go missing with more than 353 entries in chuni_score_best, but that number is a bit of a red herring, as it turns the number to trigger it can vary. I started looking into what songs would miss data, as I did find a pattern to it:

Issues only start occurring when Artemis needs to return more than 300 songs (not 300 charts).
This issue would only affect songs that had entries for more than one chart (anything that was played on only Expert or only Master never went missing). Confirmed this behavior by splitting out all of the Expert and Master scores on my main id to two different IDs, where the game would then load all of the Master/Expert scores without issue.
For songs with scores on multiple difficulties, only the first difficulty played for a given song would load (so the entry with the lowest id in chuni_score_best. For example, one song had rows with IDs 4, 412, 662 and 1005 and the game was only loading the score from the entry in row 4. So playing a new difficulty won't bring back the older scores (they are saved in the database, they just aren't being returned by Artemis).
However, songs that had all of their entries "close" (in terms of their id in `chuni_song_bests') would not run into this issue. For example, I had one song with rows with IDs of 2, 5 and 231 and none of those scores would go missing, the game would always load all three scores. This also held true for newer songs -- for example, one song with row IDs 875 and 878 would always load both scores.

From what I can tell, Chunithm requests the scores in blocks of 300 songs and songs with scores on multiple difficulties in the same "block" of IDs (0-300, 301-600, 601-900) do get returned properly, but not when scores are split across that range.

Recently noticed that scores saved on my profile for Chunithm Sun Plus on the current development build of Artemis suddenly started disappearing in-game. Looking into the issue further, the song scores were saved in `chuni_score_playlog` and `chuni_score_best` and did not look malformed, but for some songs the scores would go missing after logging out and then carding back in. Figuring I might have blown up my profile messing around, I created another one to see if there was something wrong with the game, and scores there saved fine. So I then reassigned all of the score entries in those two tables to that new user... and then the same scores disappeared. Weird. The scores are still in Artemis (and even update properly in the playlog and best tables!) but were not showing up in-game. Poking at the scores, I found that they all did load if I had cut down the list in `chuni_score_best` to a smaller number, and then at some point a bunch of scores would just disappear, all at once. Enabled debug logging in the Chunithm module to try to find out where exactly this was going wrong, starting with the smaller subset of scores, basically going in order from 0 to X in the database plus adding a few scores from later IDs that I noticed would go missing: ```254 entries in chuni_score_best 203 songs with one score, 25 with two scores, 1 song with three scores [2024-01-02 21:57:35,064] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1 [2024-01-02 21:57:35,065] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'} [2024-01-02 21:57:35,089] Chunithm | DEBUG | Response {'userId': '1', 'length': 228, 'nextIndex': -1, 'userMusicList': [snip] ``` This loaded fine. ```324 entries in chuni_score_best 235 songs with one score, 44 with two scores, 1 song with three scores [2024-01-02 22:05:21,122] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1 [2024-01-02 22:05:21,123] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'} [2024-01-02 22:05:21,157] Chunithm | DEBUG | Response {'userId': '1', 'length': 279, 'nextIndex': -1, 'userMusicList': [snip] ``` This also loaded all of the scores. ```364 entries in chuni_score_best 259 songs with one score, 48 with two scores, 2 songs with three scores [2024-01-02 22:09:53,080] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1 [2024-01-02 22:09:53,081] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '0', 'maxCount': '300'} [2024-01-02 22:09:53,119] Chunithm | DEBUG | Response {'userId': '1', 'length': 300, 'nextIndex': 300, 'userMusicList': [snip] [2024-01-02 22:09:53,206] Chunithm | INFO | v215 GetUserMusicApi request from 127.0.0.1 [2024-01-02 22:09:53,207] Chunithm | DEBUG | {'userId': '1', 'nextIndex': '300', 'maxCount': '300'} [2024-01-02 22:09:53,215] Chunithm | DEBUG | Response {'userId': '1', 'length': 8, 'nextIndex': -1, 'userMusicList': [{'length': 1, 'userMusicDetailList': [snip] ``` This is when scores went missing. It turned out from my testing that the scores would go missing with more than 353 entries in `chuni_score_best`, but that number is a bit of a red herring, as it turns the number to trigger it can vary. I started looking into what songs would miss data, as I did find a pattern to it: - Issues only start occurring when Artemis needs to return more than 300 _songs_ (not 300 charts). - This issue would only affect songs that had entries for more than one chart (anything that was played on only Expert or only Master never went missing). Confirmed this behavior by splitting out all of the Expert and Master scores on my main id to two different IDs, where the game would then load all of the Master/Expert scores without issue. - For songs with scores on multiple difficulties, only the first difficulty played for a given song would load (so the entry with the lowest id in `chuni_score_best`. For example, one song had rows with IDs 4, 412, 662 and 1005 and the game was only loading the score from the entry in row 4. So playing a new difficulty won't bring back the older scores (they are saved in the database, they just aren't being returned by Artemis). - However, songs that had all of their entries "close" (in terms of their id in `chuni_song_bests') would not run into this issue. For example, I had one song with rows with IDs of 2, 5 and 231 and none of those scores would go missing, the game would always load all three scores. This also held true for newer songs -- for example, one song with row IDs 875 and 878 would always load both scores. From what I can tell, Chunithm requests the scores in blocks of 300 songs and songs with scores on multiple difficulties in the same "block" of IDs (0-300, 301-600, 601-900) do get returned properly, but not when scores are split across that range.

Kumubou commented

2024-01-03 22:00:35 +00:00

I think I managed to find the source of the issue, in titles/chuni/base.py -- there's seemingly a block of code that goes over the resulting song list and then insert scores:

try:        
    while song_list[-1]["userMusicDetailList"][0]["musicId"] == music_detail[x + 1]["musicId"]:
        music = music_detail[x + 1]._asdict()
        music.pop("user")
        music.pop("id")
        song_list[-1]["userMusicDetailList"].append(music)
        song_list[-1]["length"] += 1
        x += 1
except IndexError:
    pass

Except the current code only looks at the last song in the list, when it really needs to be checking every song, as the entries in chuni_score_best are not guaranteed to be in any order (they're IDed as they happen on profiles).

I managed to fix this on my local instance by instead iterating over every song and every entry in chuni_score_best after the last one checked:

        for songIdx in range(len(song_list)): 
            for recordIdx in range(x+1, len(music_detail)):
                if song_list[songIdx]["userMusicDetailList"][0]["musicId"] == music_detail[recordIdx]["musicId"]:
                    music = music_detail[recordIdx]._asdict()
                    music.pop("user")
                    music.pop("id")
                    song_list[songIdx]["userMusicDetailList"].append(music)
                    song_list[songIdx]["length"] += 1

I haven't tested this beyond my local profile and a couple of test profiles, so this may still lead to things exploding.

It also looks like other parts of handle_get_user_music_api_request could be refactored, as it looks like the part where it iterates in for song in song_list doesn't seem to do anything anymore except update SCORE_BUFFER (and it seems to be adding entries that are already in there from when the song was first added to song_list, so it seems completely redundant).

I think I managed to find the source of the issue, in [titles/chuni/base.py](https://gitea.tendokyu.moe/Hay1tsme/artemis/src/commit/19baf05d7bb19971dcae189388cda1c114146a4d/titles/chuni/base.py) -- there's seemingly a block of code that goes over the resulting song list and then insert scores: ``` try: while song_list[-1]["userMusicDetailList"][0]["musicId"] == music_detail[x + 1]["musicId"]: music = music_detail[x + 1]._asdict() music.pop("user") music.pop("id") song_list[-1]["userMusicDetailList"].append(music) song_list[-1]["length"] += 1 x += 1 except IndexError: pass ``` Except the current code only looks at the last song in the list, when it really needs to be checking _every_ song, as the entries in `chuni_score_best` are not guaranteed to be in any order (they're IDed as they happen on profiles). I managed to fix this on my local instance by instead iterating over every song and every entry in `chuni_score_best` after the last one checked: ``` for songIdx in range(len(song_list)): for recordIdx in range(x+1, len(music_detail)): if song_list[songIdx]["userMusicDetailList"][0]["musicId"] == music_detail[recordIdx]["musicId"]: music = music_detail[recordIdx]._asdict() music.pop("user") music.pop("id") song_list[songIdx]["userMusicDetailList"].append(music) song_list[songIdx]["length"] += 1 ``` I haven't tested this beyond my local profile and a couple of test profiles, so this may still lead to things exploding. It also looks like other parts of `handle_get_user_music_api_request` could be refactored, as it looks like the part where it iterates in `for song in song_list` doesn't seem to do anything anymore except update `SCORE_BUFFER` (and it seems to be adding entries that are already in there from when the song was first added to `song_list`, so it seems completely redundant).

Kumubou referenced this issue

2024-01-03 22:13:47 +00:00

Fix issue in handle_get_user_music_api_request for Chunithm where songs would not always return all scores #92

Kumubou commented

2024-01-03 22:18:17 +00:00

Pull request with fix submitted in #92