Normally, there are 3 audio codecs for AI ASR and TTS, WAV(PCM) MP3 and Opus. Opus is now the default codec of WerRTC. It has some advantages for real time audio data transmission, such as short latency, automatic correction and so on.But it is also cost more CPU power. In xiaozhi-esp32 and xiaozhi-esp32-server projects, opus codecs are implemented in software. A comment in xiaozhi-esp32 project says: (In managed_components\78__esp-opus-encoder\opus_encoder.cc)

    // Complexity 5 almost uses up all CPU of ESP32C3
    SetComplexity(5);

In xiaozhi-esp32-server, it uses python lib opuslib_next to encode/decode opus. There is a test_page.html in xiaozhi-esp32-server, and it can also encode/decode opus. It uses a browser js lib libopus.js, which is in the same folder with the html file. But I can't find this lib in npm site, and this lib can't be used in node.js environment or miniprogram. This libopus.js is using webassemlby, maybe because opus needs more CPU efficiency. In Linux, opus is a 3rd lib/software just like ffmpeg.

When you full deployed xiaozhi-esp32-server on a pubic linux, you may want to enable domain name and https access. Here is how: 0. Make sure your http and ws access to your server works.

  • Use browser to access http://your_ip:8001 , It should show the login page.
  • Use browser to access ota url: http://your_ip:8002/xiaozhi/ota/, It should show the correct message.
  • Use a websocket test tool, for example: Websocket test client in chrome web store, to test ws://your_ip:8000/xiaozhi/v1/, it should return the correct anwser.
  • You can use a real Xiaozhi device to access your own server. Change the OTA url in idf.py menuconfig and build + flash it. Try to configure a network and speak to it.
  • If you don't have a real device, you can use a test page. The offical test page https://2662r3426b.vicp.fun/test/ is not useable because the page is using https and can only access wss but ws. You should use the test page under the xiaozhi-esp32-server project path main/xiaozhi-server/test/test_page.html. Use a http-server( for example: npm install -g http-server then http-server) to serve the page and access it from the browser.
  1. Install nginx and run it as service;
  2. Install certbot, generate nginx public and private certs;
# Ubuntu/Debian
sudo apt-get install certbot python3-certbot-nginx

# generate certs
sudo certbot --nginx -d diy.esp32.cn

# automatic renew
sudo certbot renew --quiet

If certbot failed to add configuration to nginx.conf, you can do it by yourself. You need to copy the certs path and paste them into nginx.conf. For example:

  ssl_certificate /etc/letsencrypt/live/yourdomain.name/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/yourdomain.name/privkey.pem;

I'll show you in the next step.

  1. Config nginx. Add the following configuration to nginx.conf, change yourdomain.name to yours:

server {
  listen 80;
  server_name yourdomain.name;
  location / {
    proxy_pass http://localhost:8001;
  }
}
server {
  listen 443 ssl;
  server_name yourdomain.name;
  ssl_certificate /etc/letsencrypt/live/yourdomain.name/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/yourdomain.name/privkey.pem;
  
  location / {
    proxy_pass https://localhost:8001;
    
    # Normal proxy header, forward client's request information.
    proxy_set_header Host localhost:8001;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Set timeout as you wish
    #proxy_read_timeout 86400s;
  }
  location = /xiaozhi/v1/ {
    proxy_pass http://localhost:8000/xiaozhi/v1/;
    # Support WebSocket
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    
    
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    #proxy_read_timeout 86400s;
  }
}

Restart nginx by nginx -s reload or service nginx restart or systemctl restart nginx.

The ota url and wss url should work now. You can try to access https://yourdomain.name/xiaozhi/ota/ and wss://yourdomain.name/xiaozhi/v1/

  1. Change vue config file main/manager-web/vue.config.js, add a element https: true, under devServer:{}, like:
  devServer: {
    port: 8001,
    https: true,
    ...

Restart npm run serve, https://yourdomain.name/ should works. and you can also access it by ip: https://your_ip:8001. The http access is disabled at the same time.

  1. Change the ota and websocket config in the admin page of https://yourdomain.name/.

Set systemd service

Set systemd service redis, manager-api, manager-web, xiaozhi-server. manager-api should rely on redis, or it will fail to start. The ExecStart should use absolute path because systemd started earlier before PATH is loaded.

/etc/systemd/system/redis.service

[Unit]
Description=Redis Server
After=network.target
Wants=network.target

[Service]
Type=simple
ExecStart=/usr/sbin/service redis start
ExecStop=/usr/sbin/service redis stop
Restart=always
RestartSec=5
StandardOutput=append:/var/log/redis.log
StandardError=append:/var/log/redis.log

[Install]
WantedBy=multi-user.target

/etc/systemd/system/manager-api.service

[Unit]
Description=Manager API Service
After=redis.service
Requires=redis.service

[Service]
Type=simple
ExecStart=/www/server/java/jdk-21.0.2/bin/java -jar /root/s2/xiaozhi-esp32-server/main/manager-api/target/xiaozhi-esp32-api.jar
WorkingDirectory=/root/s2/xiaozhi-esp32-server/main/manager-api
Restart=no
RestartSec=5
StandardOutput=append:/var/log/manager-api.log
StandardError=append:/var/log/manager-api.log
SyslogIdentifier=manager-api

[Install]
WantedBy=multi-user.target

/etc/systemd/system/manager-web.service

[Unit]
Description=Manager Web Service
After=manager-api.service
Wants=manager-api.service

[Service]
Type=simple
Environment="PATH=/usr/local/bin:/usr/bin:/bin:/root/.nvm/versions/node/v22.16.0/bin"
ExecStart=/root/.nvm/versions/node/v22.16.0/bin/npm run serve
WorkingDirectory=/root/s2/xiaozhi-esp32-server/main/manager-web
Restart=always
RestartSec=5
StandardOutput=append:/var/log/manager-web.log
StandardError=append:/var/log/manager-web.log
SyslogIdentifier=manager-web

[Install]
WantedBy=multi-user.target

/etc/systemd/system/xiaozhi-server.service

[Unit]
Description=Xiaozhi Server
After=manager-web.service
Wants=manager-web.service

[Service]
Type=simple
Environment=XIAOZHI_HOME=/root/s2/xiaozhi-esp32-server/main
ExecStart=/www/server/pyporject_evn/xiaozhi/bin/python app.py
WorkingDirectory=/root/s2/xiaozhi-esp32-server/main/xiaozhi-server
Restart=always
RestartSec=5
StandardOutput=append:/var/log/xiaozhi-server.log
StandardError=append:/var/log/xiaozhi-server.log
SyslogIdentifier=xiaozhi-server

[Install]
WantedBy=multi-user.target

enable systemd service and check logs

systemctl enable redis manager-api manager-web xiaozhi-server
reboot

Reboot and check if services are active

systemctl status redis manager-api manager-web xiaozhi-server

# Check running log
journalctl -u redis
journalctl -u manager-api
journalctl -u manager-web
journalctl -u xiaozhi-server

# Check services' stdio output log
cat /var/log/redis.log
cat /var/log/manager-api.log
cat /var/log/manager-web.log
cat /var/log/xiaozhi-server.log

I tried codebuddy in wechat dev tool, and try to make a miniprogram. I found it had a very big illusionary problem, and very proud, or very arrogant. It is always trying to make some codes without any basis or reference. I ask it to create pages, It will not just create pages but also fill codes of nonsense. I just want it to create pages and then I will provide it APIs then ask it to follow APIs to create codes. But it just create pages and codes without API documents, and the codes are just illusions and bullshits. CodeBuddy will not follow my prompts carefully, will not do want I want it to do, and will just code randomly. I'm using the default model and don't know what it is. Sonnet 3.7 is much more better, It will not create nonsense codes. CodeBuddy is much more faster than sonnet 3.7, but faster bullshit is bullshit. CodeBuddy command @ just has a very few functions. I can just add files one by one. Unlike Trae, it can't add workspace or folders.

And then I watched a video made by the codebuddy development team tencent TDP https://www.bilibili.com/video/BV1exJxzFEXW. In the video, TDP use vscode codebuddy plugin instead of wechat dev tool. And the prompt is a simple project requirement. After read the prompt, codebuddy will ask a few questions. It looks very different from the one in wechat dev tool.

The biggest problem of using AI for programming, is to understand two different bug types. One bug type is caused by documents or prompts are not detailed enough, or they are so long that AI ignored some of them, another bug type is caused by AI's ability is not enough.

To fix the first one, just give proper documents and prompts, detailed, and not too long.

To fix the second one, you need to debug by yourself. Luckily, most of these bugs are very stupid. You just need to change a few codes.

But it is diffcult to figure out which is which. A wrong decision will waste a lot of time. Maybe only you should use AI day after day, and you will know AI's temperament.