Vanilla/artemis
1
0
Fork 0
forked from Hay1tsme/artemis
Code Pull Requests Activity

Merge branch 'develop' into idac

Browse Source
This commit is contained in:
Dniel97
2023-11-13 16:18:04 +01:00
parent 4c3852d6d4 1a4dc88c8a
commit 565dc38e9a
161 changed files with 3867 additions and 1650 deletions
Download Patch File Download Diff File Expand all files Collapse all files

246
docs/INSTALL_DOCKER.md Normal file
View File

@ -0,0 +1,246 @@
# ARTEMiS - Docker Installation Guide
This step-by-step guide will allow you to install a Contenerized Version of ARTEMiS inside Docker, some steps can be skipped assuming you already have pre-requisite components and modules installed.
This guide assumes using Debian 12(bookworm-stable) as a Host Operating System for most of packages and modules.
## Pre-Requisites:
- Linux-Based Operating System (e.g. Debian, Ubuntu)
- Docker (https://get.docker.com)
- Python 3.9+
- (optional) Git
## Install Python3.9+ and Docker
```
(if this is a fresh install of the system)
sudo apt update && sudo apt upgrade
(installs python3 and pip)
sudo apt install python3 python3-pip
(installs docker)
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
(optionally install git)
sudo apt install git
```
## Get ARTEMiS
If you installed git, clone into your choice of ARTEMiS git repository, e.g.:
```
git clone <ARTEMiS Repo> <folder>
```
If not, download the source package, and unpack it to the folder of your choice.
## Prepare development/home configuration
To build our Docker setup, first we need to create some folders and copy some files around
- Create 'aime', 'configs', 'AimeDB', and 'logs' folder in ARTEMiS root folder (where all source files exist)
- Inside configs folder, create 'config' folder, and copy all .yaml files from example_config to config (thats all files without nginx_example.conf)
- Edit .yaml files inside configs/config to suit your server needs
- Edit core.yaml inside configs/config:
```
set server.listen_address: to "0.0.0.0"
set title.hostname: to machine's IP address, e.g. "192.168.x.x", depending on your network, or actual hostname if your configuration is already set for dns resolve
set database.host: to "ma.db"
set database.memcached_host: to "ma.memcached"
set aimedb.key: to "<actual AIMEDB key>"
```
## Running Docker Compose
After configuring, go to ARTEMiS root folder, and execute:
```
docker compose up -d
```
("-d" argument means detached or daemon, meaning you will regain control of your terminal and Containers will run in background)
This will start pulling and building required images from network, after it's done, a development server should be running, with server accessible under machine's IP, frontend with port 8090, and PHPMyAdmin under port 9090.
- To turn off the server, from ARTEMiS root folder, execute:
```
docker compose down
```
- If you changed some files around, and don't see your changes applied, execute:
```
(turn off the server)
docker compose down
(rebuild)
docker compose build
(turn on)
docker compose up -d
```
- If you need to see logs from containers running, execute:
```
docker compose logs
```
- add '-f' to the end if you want to follow logs.
## Running commands
If you need to execute python scripts supplied with the application, use `docker compose exec app python3 <script> <command>`, for example `docker compose exec app python3 dbutils.py version`
## Persistent DB
By default, in development mode, ARTEMiS database is stored temporarily, if you wish to keep your database saved between restarts, we need to bind the database inside the container to actual storage/folder inside our server, to do this we need to make a few changes:
- First off, edit docker-compose.yml, and uncomment 2 lines:
```
(uncomment these two)
#volumes:
# - ./AimeDB:/var/lib/mysql
```
- After that, start up the server, this time Database will be saved in AimeDB folder we created in our configuration steps.
- If you wish to save it in another folder and/or storage device, change the "./AimeDB" target folder to folder/device of your choice
NOTE (NEEDS FIX): at the moment running development mode with persistent DB will always run database creation script at the start of application, while it doesn't break database outright, it might create some issues, a temporary fix can be applied:
- Start up containers with persistent DB already enabled, let application create database
- After startup, `docker compose down` the instance
- Edit entrypoint.sh and remove the `python3 dbutils.py create` line from Development mode statement
- Execute `docker compose build` and `docker compose up -d` to rebuild the app and start the containers back
## Adding importer data
To add data using importer, we can do that a few ways:
### Use importer locally on server
For that we need actual GameData and Options supplied somehow to the server system, be it wsl2 mounting layer, a pendrive with data, network share, or a direct copy to the server storage
With python3 installed on system, install requirements.txt directly to the system, or through python3 virtual-environment (python3-venv)
Default mysql/mariadb client development packages will also be required
- In the system:
```
sudo apt install default-libmysqlclient-dev build-essential pkg-config libmemcached-dev
sudo apt install mysql-client
OR
sudo apt install libmariadb-dev
```
- In the root ARTEMiS folder
```
python3 -m pip install -r requirements.txt
```
- If we wish to layer that with python3 virtual-environment, install required system packages, then:
```
sudo apt install python3-venv
python3 -m venv /path/to/venv
cd /path/to/venv/bin
python3 -m pip install -r /path/to/artemis/requirements.txt
```
- Depending on how you installed, now you can run read.py using:
- For direct installation, from root ARTEMiS folder:
```
python3 read.py <args>
```
- Or from python3 virtual environment, from root ARTEMiS folder:
```
/path/to/python3-venv/bin/python3 /path/to/artemis/read.py <args>
```
- We need to expose database container port, so that read.py can communicate with the database, inside docker-compose.yml, uncomment 2 lines in the database container declaration (db):
```
#ports:
# - "3306:3306"
```
- Now, `docker compose down && docker compose build && docker compose up -d` to restart containers
Now to insert the data, by default, docker doesn't expose container hostnames to root system, when trying to run read.py against a container, it will Error that hostname is not available, to fix that, we can add database hostname by hand to /etc/hosts:
```
sudo <editor of your choice> /etc/hosts
add '127.0.0.1 ma.db' to the table
save and close
```
- You can remove the line in /etc/hosts and de-expose the database port after successful import (this assumes you're using Persistent DB, as restarting the container without it will clear imported data).
### Use importer on remote Linux system
Follow the system and python portion of the guide, installing required packages and python3 modules, Download the ARTEMiS source.
- Edit core.yaml and insert it into config catalog:
```
database:
host: "<hostname of target system>"
```
- Expose port 3306 from database docker container to system, and allow port 3306 through system firewall to expose port to the system from which you will be importing data. (Remember to close down the database ports after finishing!)
- Import data using read.py
### Use importer on remote Windows system
Follow the [windows](docs/INSTALL_WINDOWS.md) guide for installing python dependencies, download the ARTEMiS source.
- Edit core.yaml and insert it into config catalog:
```
database:
host: "<hostname of target system>"
```
- Expose port 3306 from database docker container to system, and allow port 3306 through system firewall to expose port to the system from which you will be importing data.
- For Windows, also allow port 3306 outside the system so that read.py can communicate with remote database. (Remember to close down the database ports after finishing!)
# Troubleshooting
## Game does not connect to ARTEMiS Allnet Server
Double check your core.yaml if all addresses are correct and ports are correctly set and/or opened.
## Game does not connect to Title Server
Title server hostname requires your actual system hostname, from which you set up the Containers, or it's IP address, you can get the IP by using command `ip a` which will list all interfaces, and one of them should be your system IP (typically under eth0).
## Unhandled command in AimeDB
Make sure you have a proper AimeDB Key added to configuration.
## Memcached Error in ARTEMiS application causes errors in loading data
Currently when running ARTEMiS from master branch, there is a small bug that causes app to always configure memcached service to 127.0.0.1, to fix that, locate cache.py file in core/data, and edit:
```
memcache = pylibmc.Client([hostname]), binary=True)
```
to:
```
memcache = pylibmc.Client(["ma.memcached"], binary=True)
```
And build the containers again.
This will fix errors loading data from server.
(This is fixed in development branch)
## read.py "Can't connect to local server through socket '/run/mysqld/mysqld.sock'"
sqlalchemy by default reads any ip based connection as socket, thus trying to connect locally, please use a hostname (such as ma.db as in guide, and do not localhost) to force it to use a network interface.
### TODO:
- Production environment

7
docs/config.md
View File

@ -6,11 +6,18 @@
- `name`: Name for the server, used by some games in their default MOTDs. Default `ARTEMiS`
- `is_develop`: Flags that the server is a development instance without a proxy standing in front of it. Setting to `False` tells the server not to listen for SSL, because the proxy should be handling all SSL-related things, among other things. Default `True`
- `threading`: Flags that `reactor.run` should be called via the `Thread` standard library. May provide a speed boost, but removes the ability to kill the server via `Ctrl + C`. Default: `False`
- `check_arcade_ip`: Checks IPs against the `arcade` table in the database, if one is defined. Default `False`
- `strict_ip_checking`: Rejects clients if there is no IP in the `arcade` table for the respective arcade
- `log_dir`: Directory to store logs. Server MUST have read and write permissions to this directory or you will have issues. Default `logs`
## Title
- `loglevel`: Logging level for the title server. Default `info`
- `hostname`: Hostname that gets sent to clients to tell them where to connect. Games must be able to connect to your server via the hostname or IP you spcify here. Note that most games will reject `localhost` or `127.0.0.1`. Default `localhost`
- `port`: Port that the title server will listen for connections on. Set to 0 to use the Allnet handler to reduce the port footprint. Default `8080`
- `port_ssl`: Port that the secure title server will listen for connections on. Set to 0 to use the Allnet handler to reduce the port footprint. Default `0`
- `ssl_key`: Location of the ssl server key for the secure title server. Ignored if `port_ssl` is set to `0` or `is_develop` set to `False`. Default `cert/title.key`
- `ssl_cert`: Location of the ssl server certificate for the secure title server. Must not be a self-signed SSL. Ignored if `port_ssl` is set to `0` or `is_develop` is set to `False`. Default `cert/title.pem`
- `reboot_start_time`: 24 hour JST time that clients will see as the start of maintenance period. Leave blank for no maintenance time. Default: ""
- `reboot_end_time`: 24 hour JST time that clients will see as the end of maintenance period. Leave blank for no maintenance time. Default: ""
## Database
- `host`: Host of the database. Default `localhost`
- `username`: Username of the account the server should connect to the database with. Default `aime`

167
docs/game_specific_info.md
View File

@ -66,6 +66,28 @@ python read.py --game SDBT --version <version ID> --binfolder /path/to/game/fold
The importer for Chunithm will import: Events, Music, Charge Items and Avatar Accesories.
### Config
Config file is located in `config/chuni.yaml`.
| Option | Info |
|------------------|----------------------------------------------------------------------------------------------------------------|
| `news_msg` | If this is set, the news at the top of the main screen will be displayed (up to Chunithm Paradise Lost) |
| `name` | If this is set, all players that are not on a team will use this one by default. |
| `rank_scale` | Scales the in-game ranking based on the number of teams within the database |
| `use_login_bonus`| This is used to enable the login bonuses |
| `crypto` | This option is used to enable the TLS Encryption |
**If you would like to use network encryption, the following will be required underneath but key, iv and hash are required:**
```yaml
crypto:
encrypted_only: False
keys:
13: ["0000000000000000000000000000000000000000000000000000000000000000", "00000000000000000000000000000000", "0000000000000000"]
```
### Database upgrade
Always make sure your database (tables) are up-to-date:
@ -93,6 +115,36 @@ After a failed Online Battle the room will be deleted. The host is used for the
- Timer countdown should be handled globally and not by one user
- Game can freeze or can crash if someone (especially the host) leaves the matchmaking
### Rivals
You can configure up to 4 rivals in Chunithm on a per-user basis. There is no UI to do this currently, so in the database, you can do this:
```sql
INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user1>, <version>, <user2>, 2);
INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user2>, <version>, <user1>, 2);
```
Note that the version **must match**, otherwise song lookup may not work.
### Teams
You can also configure teams for users to be on. There is no UI to do this currently, so in the database, you can do this:
```sql
INSERT INTO aime.chuni_profile_team (teamName) VALUES (<teamName>);
```
Team names can be regular ASCII, and they will be displayed ingame.
On smaller installations, you may also wish to enable scaled team rankings. By default, Chunithm determines team ranking within the first 100 teams. This can be configured in the YAML:
```yaml
team:
rank_scale: True # Scales the in-game ranking based on the number of teams within the database, rather than the default scale of ~100 that the game normally uses.
```
### Favorite songs
You can set the songs that will be in a user's Favorite Songs category using the following SQL entries:
```sql
INSERT INTO aime.chuni_item_favorite (user, version, favId, favKind) VALUES (<user>, <version>, <songId>, 1);
```
The songId is based on the actual ID within your version of Chunithm.
## crossbeats REV.
@ -147,18 +199,19 @@ For versions pre-dx
| SDBM | 5 | maimai ORANGE PLUS |
| SDCQ | 6 | maimai PiNK |
| SDCQ | 7 | maimai PiNK PLUS |
| SDDK | 8 | maimai MURASAKI |
| SDDK | 9 | maimai MURASAKI PLUS |
| SDDZ | 10 | maimai MILK |
| SDDZ | 11 | maimai MILK PLUS |
| SDDK | 8 | maimai MURASAKi |
| SDDK | 9 | maimai MURASAKi PLUS |
| SDDZ | 10 | maimai MiLK |
| SDDZ | 11 | maimai MiLK PLUS |
| SDEY | 12 | maimai FiNALE |
| SDEZ | 13 | maimai DX |
| SDEZ | 14 | maimai DX PLUS |
| SDEZ | 15 | maimai DX Splash |
| SDEZ | 16 | maimai DX Splash PLUS |
| SDEZ | 17 | maimai DX Universe |
| SDEZ | 18 | maimai DX Universe PLUS |
| SDEZ | 19 | maimai DX Festival |
| SDEZ | 17 | maimai DX UNiVERSE |
| SDEZ | 18 | maimai DX UNiVERSE PLUS |
| SDEZ | 19 | maimai DX FESTiVAL |
| SDEZ | 20 | maimai DX FESTiVAL PLUS |
### Importer
@ -217,6 +270,9 @@ Config file is located in `config/diva.yaml`.
| `unlock_all_modules` | Unlocks all modules (costumes) by default, if set to `False` all modules need to be purchased |
| `unlock_all_items` | Unlocks all items (customizations) by default, if set to `False` all items need to be purchased |
### Custom PV Lists (databanks)
In order to use custom PV Lists, simply drop in your .dat files inside of /titles/diva/data/ and make sure they are called PvList0.dat, PvList1.dat, PvList2.dat, PvList3.dat and PvList4.dat exactly.
### Database upgrade
@ -261,9 +317,19 @@ Config file is located in `config/ongeki.yaml`.
| Option | Info |
| ---------------- | -------------------------------------------------------------------------------------------------------------- |
| `enabled_gachas` | Enter all gacha IDs for Card Maker to work, other than default may not work due to missing cards added to them |
| `crypto` | This option is used to enable the TLS Encryption |
Note: 1149 and higher are only for Card Maker 1.35 and higher and will be ignored on lower versions.
**If you would like to use network encryption, the following will be required underneath but key, iv and hash are required:**
```yaml
crypto:
encrypted_only: False
keys:
7: ["0000000000000000000000000000000000000000000000000000000000000000", "00000000000000000000000000000000", "0000000000000000"]
```
### Database upgrade
Always make sure your database (tables) are up-to-date:
@ -272,6 +338,68 @@ Always make sure your database (tables) are up-to-date:
python dbutils.py --game SDDT upgrade
```
### Controlling Events (Ranking Event, Technical Challenge Event, Mission Event)
Events are controlled by 2 types of enabled events:
- RankingEvent (type 6), TechChallengeEvent (type 17)
- AcceptRankingEvent (type 7), AcceptTechChallengeEvent (type 18)
Both Ranking and Accept must be enabled for event to function properly
Event will run for the time specified in startDate and endDate
AcceptRankingEvent and AcceptTechChallengeEvent are reward period for events, which specify from what startDate until endDate you can collect the rewards for attending the event, so the reward period must start in the future, e.g. :
- RankingEvent startDate 2023-12-01 - endDate 2023-12-30 - period in which whole event is running
- AcceptRankingEvent startDate 2023-12-23 - endDate 2023-12-30 - period in which you can collect rewards for the event
If player misses the AcceptRankingEvent period - ranking will be invalidated and receive lowest reward from the event (typically 500x money)
Technical Challenge Song List:
Songs that are used for Technical Challenge are not stored anywhere in data files, so you need to fill the database table by yourself, you can gather all songs that should be in Technical Challenges from ONGEKI japanese wikis, or, you can create your own sets:
Database table : `ongeki_static_tech_music`
```
id: Id in table, just increment for each entry
version: version of the game you want the tech challenge to be in (from RED and up)
eventId: Id of the event in ongeki_static_events, insert the Id of the TechChallengeEvent (type 17) you want the song be assigned to
musicId: Id of the song you want to add, use songId from ongeki_static_music table
level: Difficulty of the song you want to track during the event, from 0(basic) to 3(master)
```
Current implementation of Ranking and Technical Challenge Events are updated on every profile save to the Network, and Ranked on each player login, in official specification, calculation for current rank on the network should be done in the maintenance window
Mission Event (type 13) is a monthly type of event, which is used when another event doesn't have it's own Ranking or Technical Challenge Event running, only one Mission Event should be running at a time, so enable only the specific Mission you want to run currently on the Network
If you're often trying fresh cards, registering new profiles etc., you can also consider disabling all Announcement Events (type 1), as it will disable all the banners that pop up on login (they show up only once though, so if you click through them once they won't show again)
Event type 2 in Database are Advertisement Movies, enable only 1 you want to currently play, and disable others
Present and Reward List - populate reward list using read.py
Create present for players by adding an entry in `ongeki_static_present_list`
```
id: unique for each entry
version: game version you want the present be in
presentId: id of the present - starts with 1001 and go up from that, must be unique for each reward(don't set multiple rewardIds with same presentId)
presentName: present name which will be shown on the bottom when received
rewardId: ID of item from ongeki_static_rewards
stock: how many you want to give (like 5 copies of same card, or 10000 money, etc.)
message: no idea, can be left empty
startDate: date when to start giving out
endDate: date when ends
```
After inserting present to the table, add the presentId into players `ongeki_static_item`, where itemKind is 9, itemId is the presentId, and stock set 1 and isValid to 1
After that, on next login the present should be received (or whenever it supposed to happen)
## Card Maker
### SDED
@ -284,15 +412,21 @@ python dbutils.py --game SDDT upgrade
### Support status
* Card Maker 1.30:
* CHUNITHM NEW!!: Yes
* maimai DX UNiVERSE: Yes
* O.N.G.E.K.I. bright: Yes
#### Card Maker 1.30:
* CHUNITHM NEW!!: Yes
* maimai DX UNiVERSE: Yes
* O.N.G.E.K.I. bright: Yes
* Card Maker 1.35:
* CHUNITHM SUN: Yes (NEW PLUS!! up to A032)
* maimai DX FESTiVAL: Yes (up to A035) (UNiVERSE PLUS up to A031)
* O.N.G.E.K.I. bright MEMORY: Yes
#### Card Maker 1.35:
* CHUNITHM:
* NEW!!: Yes
* NEW PLUS!!: Yes (added in A028)
* SUN: Yes (added in A032)
* maimai DX:
* UNiVERSE PLUS: Yes
* FESTiVAL: Yes (added in A031)
* FESTiVAL PLUS: Yes (added in A035)
* O.N.G.E.K.I. bright MEMORY: Yes
### Importer
@ -511,6 +645,8 @@ python dbutils.py --game SDEW upgrade
- Player title is currently static and cannot be changed in-game
- QR Card Scanning currently only load a static hero
**Network hashing in GssSite.dll must be disabled**
### Credits for SAO support:
- Midorica - Limited Network Support
@ -646,3 +782,4 @@ python dbutils.py --game SDGT upgrade
- Kinako: For helping with the timeRelease unlocking of courses and special mode
A huge thanks to all people who helped shaping this project to what it is now and don't want to be mentioned here.