aurora-admin-panel / backend Goto Github PK

View Code? Open in Web Editor NEW

20.0 20.0 26.0 197.63 MB

The backend for Aurora Admin Panel

License: MIT License

Dockerfile 0.60% Python 86.17% Mako 0.22% Shell 13.01%

backend's People

Contributors

Stargazers

Watchers

backend's Issues

优化：Fabric替换Ansible

Bug: internal error when deleting servers with forward rules

backend_1   | Sending clean.clean_runner task
backend_1   | INFO:     192.168.176.6:50466 - "DELETE /api/v1/servers/15 HTTP/1.0" 200 OK
backend_1   | INFO:     192.168.176.6:50474 - "GET /api/v2/servers?page=0&size=20 HTTP/1.0" 200 OK
backend_1   | INFO:     192.168.176.6:50498 - "GET /api/v2/servers/29/detailed HTTP/1.0" 200 OK
backend_1   | Sending clean.clean_runner task
backend_1   | INFO:     192.168.176.6:50520 - "DELETE /api/v1/servers/29 HTTP/1.0" 500 Internal Server Error
backend_1   | /usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py:1359: SAWarning: DELETE statement on table 'server_user' expected to delete 3 row(s); 0 were matched.  Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
backend_1   |   util.warn(
backend_1   | /usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py:1359: SAWarning: DELETE statement on table 'port_user' expected to delete 3 row(s); 0 were matched.  Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
backend_1   |   util.warn(
backend_1   | ERROR:    Exception in ASGI application
backend_1   | Traceback (most recent call last):
backend_1   |   File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 389, in run_asgi
backend_1   |     result = await app(self.scope, self.receive, self.send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
backend_1   |     return await self.app(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
backend_1   |     await super().__call__(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
backend_1   |     await self.middleware_stack(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
backend_1   |     raise exc from None
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
backend_1   |     await self.app(scope, receive, _send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__
backend_1   |     response = await self.dispatch_func(request, self.call_next)
backend_1   |   File "/app/app/main.py", line 73, in db_session_middleware
backend_1   |     response = await call_next(request)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next
backend_1   |     task.result()
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro
backend_1   |     await self.app(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 25, in __call__
backend_1   |     response = await self.dispatch_func(request, self.call_next)
backend_1   |   File "/app/app/main.py", line 65, in sentry_exception
backend_1   |     raise e
backend_1   |   File "/app/app/main.py", line 57, in sentry_exception
backend_1   |     response = await call_next(request)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 45, in call_next
backend_1   |     task.result()
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/base.py", line 38, in coro
backend_1   |     await self.app(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 86, in __call__
backend_1   |     await self.simple_response(scope, receive, send, request_headers=headers)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 142, in simple_response
backend_1   |     await self.app(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
backend_1   |     raise exc from None
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
backend_1   |     await self.app(scope, receive, sender)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
backend_1   |     await route.handle(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
backend_1   |     await self.app(scope, receive, send)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
backend_1   |     response = await func(request)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 209, in app
backend_1   |     response_data = await serialize_response(
backend_1   |   File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 116, in serialize_response
backend_1   |     value, errors_ = field.validate(response_content, {}, loc=("response",))
backend_1   |   File "pydantic/fields.py", line 723, in pydantic.fields.ModelField.validate
backend_1   |     v, errors = self._validate_singleton(v, values, loc, cls)
backend_1   |   File "pydantic/fields.py", line 906, in pydantic.fields.ModelField._validate_singleton
backend_1   |     return self._apply_validators(v, values, loc, cls, self.validators)
backend_1   |   File "pydantic/fields.py", line 913, in pydantic.fields.ModelField._apply_validators
backend_1   |     v = validator(cls, v, values, self, self.model_config)
backend_1   |   File "pydantic/class_validators.py", line 310, in pydantic.class_validators._generic_validator_basic.lambda12
backend_1   |     return lambda cls, v, values, field, config: validator(v)
backend_1   |   File "pydantic/main.py", line 737, in pydantic.main.BaseModel.validate
backend_1   |     return cls.from_orm(value)
backend_1   |   File "pydantic/main.py", line 629, in pydantic.main.BaseModel.from_orm
backend_1   |     values, fields_set, validation_error = validate_model(cls, obj)
backend_1   |   File "pydantic/main.py", line 1040, in pydantic.main.validate_model
backend_1   |     v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
backend_1   |   File "pydantic/fields.py", line 734, in pydantic.fields.ModelField.validate
backend_1   |     v, errors = self._validate_sequence_like(v, values, loc, cls)
backend_1   |   File "pydantic/fields.py", line 767, in pydantic.fields.ModelField._validate_sequence_like
backend_1   |     r, ee = self._validate_singleton(v_, values, v_loc, cls)
backend_1   |   File "pydantic/fields.py", line 899, in pydantic.fields.ModelField._validate_singleton
backend_1   |     value, error = field.validate(v, values, loc=loc, cls=cls)
backend_1   |   File "pydantic/fields.py", line 723, in pydantic.fields.ModelField.validate
backend_1   |     v, errors = self._validate_singleton(v, values, loc, cls)
backend_1   |   File "pydantic/fields.py", line 906, in pydantic.fields.ModelField._validate_singleton
backend_1   |     return self._apply_validators(v, values, loc, cls, self.validators)
backend_1   |   File "pydantic/fields.py", line 913, in pydantic.fields.ModelField._apply_validators
backend_1   |     v = validator(cls, v, values, self, self.model_config)
backend_1   |   File "pydantic/class_validators.py", line 310, in pydantic.class_validators._generic_validator_basic.lambda12
backend_1   |     return lambda cls, v, values, field, config: validator(v)
backend_1   |   File "pydantic/main.py", line 737, in pydantic.main.BaseModel.validate
backend_1   |     return cls.from_orm(value)
backend_1   |   File "pydantic/main.py", line 629, in pydantic.main.BaseModel.from_orm
backend_1   |     values, fields_set, validation_error = validate_model(cls, obj)
backend_1   |   File "pydantic/main.py", line 1019, in pydantic.main.validate_model
backend_1   |     value = input_data.get(field.alias, _missing)
backend_1   |   File "pydantic/utils.py", line 418, in pydantic.utils.GetterDict.get
backend_1   |     return getattr(self._obj, key, default)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__
backend_1   |     return self.impl.get(instance_state(instance), dict_)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 730, in get
backend_1   |     value = self.callable_(state, passive)
backend_1   |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/strategies.py", line 717, in _load_for_state
backend_1   |     raise orm_exc.DetachedInstanceError(
backend_1   | sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <ServerUser at 0x7f933e44a0a0> is not bound to a Session; lazy load operation of attribute 'user' cannot proceed (Background on this error at: http://sqlalche.me/e/13/bhk3)

[Bug] Systemd path not exists

[Bug] ansible_runner.run() exception

[BUG] 针对多网卡的限速策略无效

环境信息

系统信息

Linux BKVM4275716 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

网卡信息（地址信息按照环境情况已脱敏）

eth0为国内侧、eth1为国外侧

br-84ee77784cda: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.21.0.1  netmask 255.255.0.0  broadcast 172.21.255.255
        ether 02:42:89:75:38:99  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:1b:6d:77:c5  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 111.111.111.111  netmask 255.255.255.0  broadcast 111.111.111.255
        inet6 fe80::216:3eff:fe07:3c04  prefixlen 64  scopeid 0x20<link>
        ether 00:00:00:00:3c:04  txqueuelen 1000  (Ethernet)
        RX packets 52104916  bytes 8011852583 (8.0 GB)
        RX errors 0  dropped 13900  overruns 0  frame 0
        TX packets 18316065  bytes 36121122223 (36.1 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 222.222.222.222  netmask 255.255.255.0  broadcast 222.222.222.255
        inet6 fe80::216:3eff:feb4:cdb8  prefixlen 64  scopeid 0x20<link>
        ether 00:00:00:b4:cd:b8  txqueuelen 1000  (Ethernet)
        RX packets 18882336  bytes 35622782661 (35.6 GB)
        RX errors 0  dropped 97256  overruns 0  frame 0
        TX packets 15052288  bytes 4005707959 (4.0 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ifb0: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet6 fe80::4c02:4dff:fe79:435c  prefixlen 64  scopeid 0x20<link>
        ether 4e:02:4d:79:43:5c  txqueuelen 32  (Ethernet)
        RX packets 18177530  bytes 35509288006 (35.5 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 18177530  bytes 35509288006 (35.5 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 15148  bytes 4793634 (4.7 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15148  bytes 4793634 (4.7 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

路由信息

default via 222.222.222.1 dev eth1
222.222.222.0/24 dev eth1 proto kernel scope link src 222.222.222.222
111.111.111.0/24 dev eth0 proto kernel scope link src 111.111.111.111
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.21.0.0/16 dev br-84ee77784cda proto kernel scope link src 172.21.0.1 linkdown

发生场景

专线双独立IP机型，机器具备国内端IP以及国外端IP，并且将默认路由指向了国外端IP

BUG原因猜测

主要原因是backend中tc流控脚本对网卡的识别逻辑场景覆盖不足导致的，当前脚本针对单网卡的机器，工作正常，流控没有什么问题。但是，如果机器具备多张网卡，tc脚本的识别逻辑会把具备默认路由的网卡（也就是外网出口网卡）作为流控策略的目标网卡，进而导致流控无效。

一般来说，双IP机型，会将外网网卡配置为系统的默认流量出口，境内网卡作为入口。我对tc的流控了解不多，只是简单测试过，说下我的猜想吧：由于在入口测的网卡没有流控策略，数据流已经在国内侧入站，经过路由表转至境外侧网卡，因为数据已经入站，所以流控无效。

手动在入口侧网卡添加tc流控策略，可以正常生效，配合ifb网卡，可以正常控制入站与出站速率。

在单网卡机器上配置面板的流控，可以正常生效。

代码位置

https://github.com/Aurora-Admin-Panel/backend/blob/main/ansible/project/files/tc.sh#L6-L7

优化：App config完整的参数检验

feature 请求: GOST 端口控制添加 Debug 选项

在 GOST 打开Debug选项后，可以查看 GOST 的具体日志。

请问是否可以考虑添加一下到配置面板中呢？

或者能有其他方式查看到GOST的详细日志吗？

任何帮助都很谢谢！

清理：未使用的API

[Bug] The clean port task cannot be executed on time

backend/tasks/app.py

Lines 116 to 121 in 8b8bc8f

 if rule.config.get("expire_second"): 

 clean_port_runner( 

 (server_id, port_num), 

 eta=datetime.now() 

 + timedelta(seconds=rule.config.get("expire_second")), 

 )

AttributeError: 'NoneType' object has no attribute 'id'
  File "huey/api.py", line 379, in _execute
    task_value = task.execute()
  File "huey/api.py", line 772, in execute
    return func(*args, **kwargs)
  File "tasks/traffic.py", line 20, in traffic_server_runner
    finished_callback=iptables_finished_handler(server.id),

Unhandled exception in task 89a96969-0e37-4391-9fa1-a3a5be1a742d.

这个问题应该是由于删除 server 后，队列中还有 ddns 任务正在执行造成的。

ImportError: cannot import name 'soft_unicode' from 'markupsafe'

aurora-worker-1   | Traceback (most recent call last):
aurora-worker-1   |   File "/usr/local/bin/ansible-playbook", line 62, in <module>
aurora-worker-1   |     import ansible.constants as C
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/ansible/constants.py", line 12, in <module>
aurora-worker-1   |     from jinja2 import Template
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
aurora-worker-1   |     from .environment import Environment
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
aurora-worker-1   |     from .defaults import BLOCK_END_STRING
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
aurora-worker-1   |     from .filters import FILTERS as DEFAULT_FILTERS  # noqa: F401
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
aurora-worker-1   |     from markupsafe import soft_unicode
aurora-worker-1   | ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.8/site-packages/markupsafe/__init__.py)
aurora-worker-1   | Identity added: /app/ansible/priv_data_dirs/10/artifacts/0cff77bb-126f-422d-9188-c34de18d49c4/ssh_key_data (/app/ansible/priv_data_dirs/10/artifacts/0cff77bb-126f-422d-9188-c34de18d49c4/ssh_key_data)
aurora-worker-1   | Traceback (most recent call last):
aurora-worker-1   |   File "/usr/local/bin/ansible-playbook", line 62, in <module>
aurora-worker-1   |     import ansible.constants as C
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/ansible/constants.py", line 12, in <module>
aurora-worker-1   |     from jinja2 import Template
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
aurora-worker-1   |     from .environment import Environment
aurora-worker-1   |   File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>

[Bug] 自动清理任务冲突

如果设置了超时自动清理任务，会导致后来更改的任务在到期后也被清理。

复现方法：

设置 iperf3 转发规则，超时时间设置 10 分钟
将转发规则从 iperf3 换成其他的转发方式，如 iptables
该端口的转发规则会在 10 分钟后被清理

考虑将超时任务的关闭检测改成 huey 周期性检查任务，定时扫描数据库里面的转发规则，如果超过了超时时间（ expire_second 在数据库里面存为到期时间戳），则执行清理任务。

	if rule.config.get("expire_second"):
	clean_port_runner(
	(server_id, port_num),
	eta=datetime.now()
	+ timedelta(seconds=rule.config.get("expire_second")),
	)