What is a key weakness of data-driven track-based prediction?
A. Over-reliance on hand-engineered physical models.
B. Inability to use HD Map context.
C. High computational cost and low explainability.
D. Cannot handle partial observation scenarios.
C.
For high computational cost, unlike simple kinematic models like Constant Velocity or Constant Steering Angle, data-driven models, such as LSTMs, Transformers, or Graph Neural Networks, require substantial memory and processing power. In the context of an autonomous vehicle, where predictions must be updated every few milliseconds, this "compute tax" is a major hurdle.
For low explainability, data-driven models operates as black boxes. If a model predics that a cyclist will suddenly veer into traffic, it is often impossible to trace the specific logic behind that decision within the millions of neural network weights. This makes it difficult to validate for safety-critical applications compared to model-based approaches where you can see the physical vectors.
Why option A is inccorect? Over-reliance on hand-engineered physical models, this is actually a weakness of model-based prediction, not data-driven prediction. Data-driven models specifically aim to move away from hand-engineered rules.
Why option B is inccorect? Inability to use HD Map context. Many data-driven tracked-based models like VectorNet are specifically designed to fuse agent tracks with HD map data. It is a strength of modern data-driven architectures, not a inherent weakness.
Why option D is inccorect? Cannot handle partial observation scenarios. Modern data-driven models are actually quite robust at "filling in the gaps" using learned temporal patterns via Recurrent Neural Networks or Attention mechanisms, often performing better than rigid physical models when sensor data is noisy or occluded.
What is a key advantage of the linear quadratic regulator (LQR) compared to PID control?
A. Requires no tuning of parameters.
B. Optimizeds a quadratic cost function for linear systems (optimal control).
C. Works without a mathematical model of the system.
D. Is computationally cheaper for real-time deployment.
B.
While PID control is a model-free apprach based on trial and error (tuning gains), LQR is a model-based approach that uses the physics of the system to find the mathematically best possible control strategy.
LQR is the gold standard for Optimal Control. It works by minimizing a specific quadratic cost function $J$.
\[J = \int_{0}^{\infty} (x^T Q x + u^T R u)dt\]$x^TQx$ represents the penalty for the state being away from the target (error).
$u^TRu$ represents the penalty for using control effort (energy/actuation).
By adjusting the $Q$ and $R$ matrices, an engineer can mathematically decide the exact trade-off between how fast the system reaches the target versus how much energy it consumes. PID control cannot provide this "guaranteed optimality" because it does not know the physics of the system.
Why other choices are not correct?
For A, requires no tuning of parameters, this is false. While we do not tune $P$, $I$, and $D$ gains directly, we must tune the weighting matrices ($Q$ and $R$).
For C, works without a mathematical model, this is inccorect. LQR requires a precise State-Space Model (in the form of $\dot{x} = Ax + Bu$). PID is the one that can work without a model.
For D, is computational cheaper, which is generally false. PID is a very simple calculation. LQR requires solving the Algebraic Riccati Equation and performing matrix multiplications online. While modern computers handle both easily, LQR is technically more complex.
In summary,
| Feature | PID Control | LQR Control |
|---|---|---|
| Model Needed? | No (Model-free) | Yes (State-space model) |
| Optimality? | No (Heuristic) | Yes (Mathematically optimal) |
| Handling MIMO? | Difficult (needs multiple loops) | Natural (handles multiple inputs/outputs) |
| Complexity | Low | Moderate |
In closed-loop learning for planning, what is the key advantage over open-loop learning?
A. No need for expert demonstration data.
B. Eliminates distribution shift between training and inference.
C. Lower computational cost during training.
D. Does not require simulation or real-world deployment.
B.
Eliminates distribution shift between training and inference. This is a fundamental concept in imitation learning and autonomous robotics. The "loop" refers to whether the model's actions affect the next state it observes during the learning process.
In open-loop learning, a model is trained on a fixed dataset of expert demonstrations. The model learns: "In state $s$, do action $a$."
However, at ingerence time, the model will inevitably make a tiny mistake. Because it is in a "closed loop" with the real world, that mistake leads to a slightly different state that what was in the training data. These errors compound quickly.
For choice B, closed-loop learning such as Reinforcement Learning or DAgger allows the model to interact with the environment during training. It exxperiences the consequences of its own actions. It learns how to recover from its own mistakes. By training on the states it actually visits rather than just the expert's perfect states the "training distribution" matches the "inference distribution," effectively eliminating the shift.
For choice A, no need for expert demonstration data. It is incorrect. Many closed-loop methods, like DAgger, specifically rely on an expert to "label" the new states the model visits.
For choice C, lower computational cost during training. It is also incorrect. Closed-loop learning is significantly more expensive because you have to constantly run a simulator or the real robot and potentally query an expert for new labels.
For choice D, does not require simulation or real-world deployment. This is the opposite of the truth. A simulator or a real-world environment is a must to close the loop; otherwise, it may stack with a static (open-loop) dataset.
What is a core challenge of using Reinforcement Learning (RL) for real-world autonomous driving planning?
A. Cannot incorporate expert demonstration data.
B. Prohibitive sample inefficiency and safety risks.
C. Lacks interpretability compared to rule-based methods.
D. Cannot handle dynamic obstacle interactions.
B.
While Reinforcement Learning (RL) has shown incredible success in virtual environments, transitioning that success to the "real-world" roads is difficult because of how RL agents learn.
Why Choice B is correct? Reiniforcement Learning is fundatmentally based on trial and error. In an autonomous driving context, this leads two massive roadblocks.
One is sample inefficiency. To learn a complex behavior, like navigating a busy intersection, an RL agent might need millions of training episodes. In the real world, we cannot afford the time of the fleet size required to gather that much data through "discovery".
The other one is safety risks (the exploration problem). For an RL agent to learn that "crashing into a wall is bad," it technically has to "explore" that action or get very close to it to receive a negative reward. In a simulator, this is fine; in a 4000-pound vehicle on a public road, "exploring" a collision is catastrophic. This makes safe exploration one of the biggest hurdles in the field.
Why other choices are incorrect?
For A, cannot incorporate expert demonstration data. This is false. Techniques like Imitation Learning or Pre-training with Expert Demonstrations are frequently used to kickstart an RL agent so it does not start from zero.
For C, lacks interpretaility compared to rule-based methods. While true that RL is a "black box" compared to "if-then" rules, this is a general weakness of all deep learning methods. Choice B is considered a more specific core challenge for deployment because even a "black box" that worked perfectly would be deployed if it weren't so dangerous and slow to train.
For D, cannot handle dynamic obstacle interactions. This is incorrect. One of the main reasons researchers want to use RL is that it is actually better at handling complex, dynamic interactions than rigid, rule-based systems. The problem is not the capability, it is the training process.
The "Sim-to-REal" Gap
To solve these issues, researchers often train RL agents in high-fidelity simulators. However, this introduces the Reality Gap, a policy that learns to drive perfectly in a simulation may fail in the real world because the simulator's physics or sensor noise is not 100% accurate.
In camera-based 3D perception, what is the core idea of the "Lift to 3D at Input" approach?
A. Convert image to pseudo-LiDAR via dense depth prediction.
B. Re-parameterize output of off-the-shelf 2D detectors.
C. Project 2D feature map to 3D/BEV space via camera intrinsic/extrinsic
D. Detect objects in 2D first then localize them in 3D.
C.
The correct answer is C. Project 2D feature map to 3D/BEV space via camera intrinsic/extrinsic.
This approach is the foundation of modern Bird's Eye View (BEV) perception stacks, famouly popularized by architectures like LSS (Lift, Splat, Shoot). Instead of detecting objects in individual images and trying to "switch" them together later, the model transforms the raw visual data into a 3D-consistent space early in the pipeline.
The "code idea" is a geometric transformation. Because 2D images lack depth, the model must "lift" the 2D features into 3D. This involves:
where $\mathbf{X}$ is the 3D coordinate and $(u, v)$ is the image coordinate.
Why the others are incorrect?
For A, convert image to pseudo-LiDAR. THis is specifically the Pseudo-LiDAR paradigm. While it is a form of lifting, it focusses on creating a point cloud from depth maps rather than projecting features maps into a unified latent space. It is considered a subset/precursor to the more general feature-lifting in Choice C.
For B, Re-parameterize output of 2D detectors. This is known as "Lifting at the Output." Models like FCOS3D detect objects in 2D first and then use geometric constraints to "guess" the 3D bounding box dimensions. This is the opposite of Choice C, which lifts at the input/feature stage.
For D, dtect objects in 2D first. This refers to "2D-Driven 3D Detection". It relies on a 2D crop to narrow down the search space in 3D, rather than creating a global 3D representation from the start.
Key Advantage
By lifting to 3D at the input/feature level, the system can naturally fuse data from multiple cameras into a single, seamless top-down view. This eliminates the "double-counting" of objects that appear in the overlap between two camera feeds.
What is a major disadvantage of Model Predictive Control (MPC) for autonomous driving?
A. Cannot handle hard constraints (e.g. max steering rate)
B. Slower than traditional methods (e.g. PID) due to optimization
C. Requires no vehicle model to operate
D. Fails to minimize tracking errors for dynamic trajectories
B.
Slower than traditional methods (e.g. PID) due to optimization. Model Predictive Control (MPC) is incredibly powerful, but its "brain" has to work much harder than a PID controller or a simple Pure Pursuit algorithm.
Unlike PID, which simply reacts to the current error, MPC is an online optimizer. At every single time step, the vehicle must:
If you have a complex non-linear vehicle model or a long prediction horizon, this requires significant CPU/GPU power. In the high-spped world of autonomous driving, if the optimization takes too long to solve, the "optimal" command arrives too late to be useful.
Why the others are incorrect?
For A, cannot handle hard constraints. This is actually the main advantage of MPC. It is specifically designed to respect physical limits, like "don't turn the steering wheel faster than 40 degrees/sec" or "don't exceed 65 mph."
For C, requires no vehicle model to operate. This is incorrect. MPC is a model-based controller. It absolutely requires a mathematical model (kinematic or dynamic) to predict how the car will behave in the future.
For D, fails to minimize tracking errors. Incorrect. MPC is generally better at minimizing tracking errors than PID because it "sees" the curves comming in the trajectory and can start turning on the path perfectly.
Summary, the MPC trade-off.
Think of PID as a driver who only looks at the road 1 foot in front of the bumper, fast and simple, but twitchy. Think of MPC as a driver looking 100 feet ahead and calculating the perfect line, smooth and safe, but requires a lot of "mental" effort to process.
What is the key function of the Derivative (D) element in a PID controller?
A. Amplify the current error to speed up response.
B. Correct steady-state errors to speed up response.
C. Introduce damping to reduce overshoot and oscillation.
D. Switch abruptly between two control states.
C.
While the Proportional (P) term looks at the present and the Integral (I) term looks at the past, the Derivative (D) term is the "prophet" of the PID controller, it looks at the future.
The Derivative term calculates the rate of change of the error. Its primary job is to act as a "brake" or a shock absorber for the system.
Why the others are incorrect?
For A, amplify the current error. THis is the job of the Proportional (P) term. A higher P-gain makes the system more aggresive in the present but oftern leads to overshoot.
For B, correct steady-state errors. This is the job of the Integral (I) term. The I term accumulates small errors over time to ensure the system eventually reaches the exact target, even if there is a constant bias (like wind pushing against a drone).
For D, switch abruptly between two states. This describes a Bang-Bang controller (like a basic thermostat that is either 100% ON or 100% OFF), not a PID controller, which provides a smooth, continuous output.
The "D" Trade-off: Noise
While the D term is great for stability, its biggest weakness is high-frequency noise. Sins the derivative is the slope of the error, a tiny bit of "jitter" in a sensor reading can result in a massive, spikey derivative. This is why, in real-world applications, the D-term is almost always paired with a low-pass filter to keep the "brakes" from stuttering.
The standard PID formula in the time domain is:
\[u(t) = K_p e(t) + K_i \int_{0}^{t} e(\tau) d \tau + K_d \frac{de(t)}{dt}\]Compare the core differences between GNSS-based and Environment Perception-Based localization in update rate and drift characteristics, and analyze the impact of these differences on the real-time control of autonomous driving.
| Feature | GNSS | Environment Perception |
|---|---|---|
| Update Rate | Low (Typically 1Hz - 10Hz) | High (Typically 10Hz - 100Hz) |
| Drift | Zero cumulative drift (Global reference) | Cumulative drift (Incremental relative motion) |
| Accuracy | Decimeters (RTK) to Meters (Standard) | Centimeters (Relative to surroundings) |
| Environment | Needs clear sky (Fails in tunnels/canyons) | Needs features (Fails in fog/featureless roads) |
GNSS: The "Noisy Truth"
GNSS provides a global coordinate. Its greatest strength is that it does not "drift." If one is at Point A, and he drive 100 miles to Point B, the error at Point B is roughly the same as it was at Point A.
Environment Perception: The "Smooth Liar"
EPB methods (like LiDAR mapping or Visual Odometry) compare current sensor data to previous frames or a pre-built HD map.
Impact on Real-Time Control
The differences between these two systems dictate how the vehicle actually "feels" on the road.
The "Control Latency" Problem
Vehicle controllers (like the PID or MPC mentioned earlier) usually require feedback at 50Hz to 100Hz to maintain stability, especially at high speeds.
Glabal vs. Local Consistency
Explain the three main approaches of "Lift to 3D" in camera-based 3D perception (Output/Input/Feature Map), and list one strength for each approach.
In camera-based 3D perception, "lifting" refers to the mathematical and architectural process of recovering the missing depth dimension to transform 2D image data into a 3D spatial representation. The three approaches differ primarily in where in the neural network pipeline this transformation occurs.
1. Lift at Output (Image-to-3D)
In this approach, the network processes the image entirely in 2D using standard convolutional backbones. The "lifting" happens at the very end (the prediction heads), where the model attempts to regress 3D properties directly from 2D features.
2. Lift at Input (Pseudo-LiDAR)
This approach transforms the 2D data into a 3D representation before the main detection of perception task begins. It essentially tries to make a camera "act" like a LiDAR sensor.
3. Lift at Feature Map (BEV/Latent Space)
This is the state-of-the-art approach used by systems like Tesla's FSD and various BEV (Bird's Eye View) models. It lifts intermediate latent features rather than raw pixels or final detections.
Summary Table
| Approach | Where Lifting Occurs | Core Mechanism | Main Strength |
|---|---|---|---|
| Output | Prediction Heads | Regression of 3D attributes | Speed & Efficiency |
| Input | Pre-processing | Dense Depth $\rightarrow$ Point Cloud | Physical Interpretability |
| Feature Map | intermediate Layers | Geometric Projection to BEV | Seamless Multi-Camera Fusion |
What are affordances in motion planning, and what are their key advantages and limitations compared to direct actuation output (steering/acceleration)?
In the context of motion planning, affordances are intermediate, high-level features that represent the "state of the world" in a way that is directly relevant to driving tasks.
Instead of a model trying to go straight from raw camera pixels to a steering wheel angle (End-to-End) or trying to identify every single pixel in the scene (Semantic Segmentation), it extracts specific "labels" that matter for decision-making.
What do they look like?
Typical affordances in autonomous driving include:
Advantages Over Direct Actuation (End-to-End)
The "Direct Acuation" approach, where a neural network takes an image and outputs a torque value, is often called a black box. Affordances offer a more transparent middle ground.
Limitations of Affordances
While they sound like the perfect compromise, affordances introduce their own set of "robotic" headaches.
Information Bottlenecks: By condensing a rich, 4K camera feed into just 10 or 15 numbers (affordances), you might throw away the "reason" for an action. For example, if there is a massive pothole in the lane, a standard affordance model might not have a "distance to pothole" label, forcing the car to drive right through it because it only cares about "distance to lane center."
Cascading Errors: If the affordance extractor makes a mistake (e.g., misidentifying a guardrail as a lane line), the motion planner will perfectly execute a "safe" path right into the rail. In direct actuation, the model might implicitly learn to avoid the visual texture of a guardrail.
Manual Engineering: Someone has to decide which affordances matter. If you forget to include "pedestrian intent" as an affordance, your planner will be "blind" to the person about to step off the curb until they are physically in the way.
Comparison Summary
| Feature | Direct Actuation (End-to-End) | Affordance-Based Planning |
|---|---|---|
| Logic | "See pixel, move wheel" | "See world, calculate distances, move wheel" |
| Transparency | Low (Black Box) | High (Explainable) |
| Generalization | Specific to vehicle dynamics | Generic across different vehicles |
| Failure Mode | Unpredictable "weird" behaviors | Predictable but limited by labels |
Given the pseudocode of the traditional CV lane detection algorithm, answer the following questions:
(1) What is the purpose of the ROI masking step? (2) Which line converts edge pixels into detected lane segments?
Algorithm 1 CV Lane Detection
Input: RGB Image $I \in \mathbb{R}^{H \times W \times 3}$
Parameters: Gaussian kernel $k$, Canny thresholds $T$, ROI vertices $V$, Hough parameters $\rho$
Output: Set of detected lane segments $\mathcal{L} = {(x_1, y_1, x_2, y_2)}$
1 $G \leftarrow \text{ConvertToGrayscale}(I)$
2 $B \leftarrow \text{GaussianBlur}(G, k)$
3 $E \leftarrow \text{CannyEdgeDetection}(B, T)$
4 $M_{roi} \leftarrow \text{GenerateMask}(V, shape(E))$
5 $E_{masked} \leftarrow E \bigodot M_{roi}$
6 $\mathcal{L} \leftarrow \text{HoughLineTransform}(E_{masked}, \rho)$
7 $\text{return } \mathcal{L}$
(1) Purpose of the ROI Masking Step
Th Region of Interest (ROI) masking step is designed to eliminate visual noise and irrelecant data from the image.
In a standard driving scene, the top half of the frame typically contains the sky, trees, or buildings. While these areas may contain "edges" (captured by the Canny detector in Line 3), they are not relevant to lane tracking. By applying the mask $M_{roi}$ via element-wise multiplication ($\bigodot$).
(2) Conversion of Edge Pixels into Lane Segments
The line that performs this conversion is Line 6:
\[\mathcal{L} \leftarrow \text{HoughLineTransform}(E_{masked}, \rho)\]Until this point in the algorithm, the data consists of a "mask" or a "map" of pixels, $E_{masked}$ is essentially a binary image where $1$ is an edge and $0$ is nothing. The computer doesn't "know" these are lines yet, it just sees a collection of scattered points.
The Hough Line Transform is the mathematical bridge. It uses a voting procedure in a parameter space (Hough Space) to determine which sets of edge pixels align into a coherent linear shape. It then outputs the coordinates ($\mathcal{L}$) of the start and end points of those segments, effectively turning "dots on a grid" into "vector geometry".
]]>Updated 3 May 2026.
The AI lifecycle: Data Collection, Model Training / Fine Tuning, Model Deployment, Model Use.
An AI security threat model contains that: Attacker's goals (AI security properties), for example, privacy, security, safety, transparency, fairness; Attacker's capability, for example, blackbox attacks and whitebox attacks; Attacker's techniques.
A classifier $f(x; \theta)$ is trained on a dataset $D \sim P_{\text{train}}$, but deployed in an environment where inputs follow a different distribution $P_{\text{adv}}$, potentially influenced by an adversary.
The generalized gap is defined as:
\[\mathbb{E}_{(x, y) \sim P_{\text{adv}}} [\mathcal{L}(f(x; \theta), y)] - \mathbb{E}_{(x, y) \sim P_{\text{train}}} [\mathcal{L}(f(x; \theta), y)]\](i) Explain how overfitting increases the generalization gap under adversarial distribution shift.
Explanation:
Overfitting occurs when a model $f(x; \theta)$ learns the noise or spurious correlations sepcific to $P_{\text{train}}$ rather than the underlying signal. Under an adversarial shift $P_{\text{adv}}$, this becomes a massive liability.
即係，過擬合係指模型學得太過死板以致於記住咗 $P_{\text{train}}$ 之中嘅噪聲或者啱啱好嘅特徵，而並非真正嘅邏輯。對抗性分布 $P_{\text{adv}}$ 之下會令到 generalization gap 極劇升高。
Overfitting causes the model to learn patterns specific to $P_{\text{train}}$, including noise, rather than generalizable features. When evaluated on $P_{\text{adv}}$, these patterns do not hold, resulting in increased loss and a larger generalization gap.
(ii) Robust optimization:
\[\min_{\theta} \max_{\delta \in \Delta} \mathcal{L}(f(x + \delta; \theta), y)\]is a framework that explicitly addresses overfitting. Explain the purpose of the inner maximization and the outer minimization, respectively. Justify why this reduces overfitting.
Explanation:
The objective $\min_{\theta} \max_{\delta \in \Delta} \mathcal{L}(f(x + \delta; \theta), y)$ is a Zero-Sum Game between an attacker and a defender.
The Inner Maximization $\max_{\delta}$. This represents the Attacker. Its goal is to find the "worst-case" perturbation $\delta$ within a defined constraint $\Delta$ (like an $\ell_{p}$ ball) that maximizes the loss. It essentially looks for the model's weakest point near every data point.
The Outer Minimization $\min_{\theta}$. This represents the Learner or Defender. Its goal is to find parameters $\theta$ that minimize the loss even when the attacker does their worst.
Why it reduces overfitting is that the robust optimization acts as a powerful form of regularization. Instead of fitting a single point $(x, y)$, the model is forced to fit an entire "neighborhood" $(x + \Delta, y)$. This prevents the decision boudary from being too close to the data points or being too jagged, as it must remain constant over the entire perturbation set $\Delta$. It shifts the model's focus from "brittle" high-frequency features to "robust" features that remain stable even under attack.
The inner maximization models adversarial perturbations, while the outer minimization is the regular loss optimization. This reduces overfitting by smoother decision boundaries.
Given a ML model with a decision function $f: X \rightarrow Y$ that maps an input sample $x \in X$ to a true class label $y_{tree} \in Y$, $x' = x + \delta$ is called an adversarial sample with an adversarial perturbation $\delta$ if $f(x') = y' \neq y_{true}$, $\vert\vert\delta\vert\vert < \epsilon$, where $\vert\vert.\vert\vert$ is a distance metric and $\epsilon$ is the maximum allowable perturbation that results in misclassification while preserving semantic integrity of $x$.
DNN Training: $\min_{\theta} \sum_{(x_i, y_i) \in D_{train}} L(f_{\theta}(x_i), y_i)$
Adversarial Attack: $\max_{x'} L(f_{\theta}(x'), y)$ subject to $\vert\vert x' - x\vert\vert_p \leq \epsilon$ for $x \in D_{test}$.
Small perturbation: $\vert\vert x' - x\vert\vert \frac{8}{255}$
The basic idea is: given an input, add non-random perturbation that induces classification mistake.
Attacker's knowledge: Loss
Attacker's knowledge: Logit
The key idea is to perturb input in the direction of the sign of the gradient of the loss w.r.t. the input, scaled by a perturbation budget.
\[x' = x + \epsilon \cdot sign(\nabla_x J(f(x), y_{true}))\]FGSM but perturbation is done in multiple smaller steps.
\[x_{(i)} = clip(x_{(i - 1)} + \alpha \cdot sign(\nabla_x J(f(x_{(i - 1)}), y_{true})))\]Target attack is given an input $x$ and a target lable $t$, the attack solves $\min_{\delta} {\vert\vert\delta\vert\vert}_2$ so that $f(x + \delta) = t$.
However the constraint is hard to enforce L-BFGS attack relaxes it to $\min_{\delta} c \cdot {\vert\vert\delta\vert\vert}_2^2 + J(f(x + \delta), t)$. Solving this optimization problem means we find an $x'$ which is close to $x$ and its loss to lable $t$ is small.
L-BFGS Attack minimize distance and loss, loss does not directly represent misclassification and maybe flip the label.
C&W attack operates on logits $Z(x)$.
\[\min_{\delta} c \cdot {\vert\vert\delta\vert\vert}_2^2 + (\max_{i \neq t} Z_i (x + \delta) - Z_t (x + \delta))\](Target class $t$)
\[\min_{\delta} c \cdot {\vert\vert\delta\vert\vert}_2^2 + (\max_{i \neq y} Z_i (x + \delta) - Z_y (x + \delta))\](Untargetted attack)
Directly enforces a decision boudary margin.
The basic idea is search.
Attacker's knowledge: confidence scores
Attacker's knowledge: predicted labels
Main defense mechanisms against adversarial examples are listed below:
Insight: Most attacks are based on the model's gradient information. Therefore, gradient masking methods hide the gradient of the model from being used by an adversary. Shattered Gradients Methods and Stochastic/Randomized Gradients Methods are the approaches.
Goal: To prevent the flow of information from the inputs to the outputs in the model, by preventing the attacker from calculating the gradients.
How: Applying a non-smooth or non-differentiable preprocessor $g(.)$ to the inputs, and then training a DNN model $f$ on the preprocessed inputs $g(x)$.
Why: The trained target classifier $f(g(x))$ is not differentiable w.r.t. the inputs $x$, causing the failure of adversarial attacks.
Thermometer Encoding's idea is to convert the continuous space (of input) to discrete one. It applies discretization of the intensity level of each pixel into an $l$-dimensional vector.
The target classifier is trained using discrete vectors for all pixels, which breaks the calculation of the gradients.
Why does it not affect training (gradient descent)?
Goal: To apply some form of randomization of the DNN model as a defense strategy to fool the advesary.
How: Add randomness, e.g., training a set of classifiers, and during the testing phase randomly selecting one classifier to predict the class labels.
Why: Because the adversary does not know which model was used for prediction, the attack success rate is reduced.
Adversarial training trains the model on adversarial examples so that it learns to classify them correctly.
Approximated by alternating optimization, where the inner maximization (attack) and the outer minimization (training) are interleaved at every minibatch.
(i) Assume each image in a dataset is a $8 \times 8$ pixel grayscale image. Each pixel in an image is denoted by a single integer between 0 and 31. Given an image, we construct its advesarial example within $l_2$ ball. Assume the budget $\epsilon = 2$, what is the maximum number of images within the $l_2$ balls? Show your working. Set up the calculation; you do not need to calculate the final value.
Solution:
An image can be represented as a vector $(X = (x_1, x_2, …, x_{64}))$ in a 64-dimensional space,where each pixel $x_i$, is an integer such that $0 \neq x_i \neq 31$.
An adversarial example $X'$ within the $l_2$ ball of radius $\epsilon$ satificies:
\[{\vert\vert X' - X \vert\vert}_2 \neq \epsilon\]Substituting the given values ($\epsilon = 2$ and 64 pixels):
\[\sqrt{\sum_{i = 1}^{64} (x_i' - x_i)^2} \leq 2\]To maximize the number of valid images $X'$, we must choose a base image $X$ such that the perturbation $\delta_i$ are not restricted by the image boundaries ($0$ and $31$). If a pixel $x_i$ is $0$, then $\delta_i$ cannot be negative; if $x_i$ is $31$, $\delta_i$ cannot be positive.
The maximum number of images is achieved when $X$ is in the "middle" of the range (e.g., $2 \le x_i \le 29$ for all $i$), allowing $\delta_i$ to take any value in ${ -2, -1, 0, 1, 2 }$ as long as the sum of squares is $\le 4$.
We need to find all integer vectors $(\delta_1, \dots, \delta_{64})$ such that $\sum \delta_i^2 \le 4$. The possible values for $\delta_i^2$ are $0, 1^2=1,$ and $2^2=4$. Any $\delta_i^2 > 4$ is impossible.
We categorize the solutions based on the value of the sum $S = \sum \delta_i^2$:
Case 1: $S = 0$All $\delta_i = 0$.
Case 2: $S = 1$
Case 3: $S = 2$
Case 4: $S = 3$
Case 5: $S = 4$
Summing all the cases above, the maximum number of images $N$ is:
\[N = 1 + \left[ \binom{64}{1} \cdot 2 \right] + \left[ \binom{64}{2} \cdot 2^2 \right] + \left[ \binom{64}{3} \cdot 2^3 \right] + \left[ \binom{64}{4} \cdot 2^4 \right] + \left[ \binom{64}{1} \cdot 2 \right]\]Combining the $\binom{64}{1}$ terms, the simplified setup is:
\[N = 1 + 4\binom{64}{1} + 4\binom{64}{2} + 8\binom{64}{3} + 16\binom{64}{4}\](ii) A classifier $f(x)$ is trained using a modified loss function:
\[\mathcal{L}'(x, y) = \mathcal{L}(f(x), y) + \lambda {\vert\vert x \vert\vert_x}^2\]where $\lambda > 0$ is a regularization parameter applied directly to the input. An adversary constructs adversarial examples using the Fast Gradient Sign Method (FGSM):
\[x_{adv} = x + \epsilon \cdot sign(\nabla_x \mathcal{L}'(x, y))\]Derive an expression for $\nabla_x \mathcal{L}' (x, y)$. Explain how the additional term $\lambda {\vert\vert x \vert\vert}_2^2$ changes the direction of the FGSM perturbation. Does this modification weaken FGSM attacks.
Solution:
We start with the modified loss function:
\[\mathcal{L}'(x, y) = \mathcal{L}(f(x), y) + \lambda {\vert\vert x \vert\vert}_2^2\]To find the gradient with respect to the input $x$, we apply the linearity of the gradient operator:
\[\nabla_x \mathcal{L}'(x, y) = \nabla_x \mathcal{L}(f(x), y) + \nabla_x (\lambda {\vert\vert x \vert\vert}_2^2)\]Recall that for the squared $L_2$ norm, ${\vert\vert x \vert\vert}_2^2 = \sum x_i^2$. The derivative of each component is $2x_i$, which gives us $\nabla_x {\vert\vert x \vert\vert}_2^2 = 2x$.
Thus, the final expression for the gradient is:
\[\nabla_x \mathcal{L}'(x, y) = \nabla_x \mathcal{L}(f(x), y) + 2\lambda x\]The standard FGSM attack moves the input in the direction of the sign of the loss gradient to maximize error. With the modification, the perturbation vector becomes:
\[x_{adv} = x + \epsilon \cdot sign\left( \underbrace{\nabla_x \mathcal{L}(f(x), y)}_{\text{Classification Gradient}} + \underbrace{2\lambda x}_{\text{Input Penalty}} \right)\]The additional term $2\lambda x$ acts as a centripetal force pulling the gradient toward the direction of the input vector itself. Here is how it changes the direction:
Yes, theoretically, but with significant caveats.
However, it is generally considered a weak defense for two reasons:
Interval Arithmetic
What is Affine Transformation?
What is ReLU Transformation?
ReluVal
What are these concepts?
Consider a ReLU-activated neural network $f(x)$:
\[f(x) = W_2 \cdot \text{ReLU}(W_1 \cdot x + b_1) + b_2\]where $W_1 \in \mathbb{R}^{3 \times 3}$, $b_1 \in \mathbb{R}^3$, $W_2 \in \mathbb{R}^{1 \times 3}$, $b_2 \in \mathbb{R}$.
Assume the first layer of $f(x)$ as
\[W_1 = \begin{bmatrix} 2 & 1 & 0 \\ 1 & -2 & 2 \\ 2 & -1 & 2 \end{bmatrix} \text{, } b_1 = \begin{bmatrix} 0, -0.5, -0.3 \end{bmatrix}^{T}\]Given an input $x \in \mathbb{R}^3$ with $0.3 \leq x_1 \leq 0.5$, and $0.3 \leq x_3 \leq 0.7$, find the bounds of the output of $z = W_1 \cdot x + b_1$ (linear layer only) using interval arithmetic.
Solution:
We have
\[z = W_1 x + b_1\]where
\[W_1 = \begin{bmatrix} 2 & 1 & 0 \\ 1 & -2 & 2 \\ 2 & -1 & 2 \end{bmatrix}, \quad b_1 = \begin{bmatrix} 0 \ -0.5 \ -0.3 \end{bmatrix}.\]Input bounds are given only for $x_1$ and $x_3$:
$x_1 \in [0.3, 0.5]$
$x_3 \in [0.3, 0.7]$
No bounds are given for $x_2$. Since only partial bounds are provided and no further assumptions are stated, we must assume $x_2$ is unbounded (can be any real number).
Let us express each component of $z$:
First component:
\[z_1 = 2x_1 + 1x_2 + 0x_3 + 0 = 2x_1 + x_2.\]Since $x_1$ is bounded but $x_2$ is unbounded,
\[z_1 \in (-\infty, \infty).\]Second component:
\[z_2 = 1x_1 - 2x_2 + 2x_3 - 0.5 = x_1 - 2x_2 + 2x_3 - 0.5.\]Again, because $x_2$ is unbounded, $z_2$ is also unbounded:
\[z_2 \in (-\infty, \infty).\]Third component:
\[z_3 = 2x_1 - 1x_2 + 2x_3 - 0.3 = 2x_1 - x_2 + 2x_3 - 0.3.\]Same situation: the presence of $x_2$ with coefficient $-1$ makes this unbounded.
Thus, without bounds on $x_2$, interval arithmetic gives:
\[\boxed{(-\infty, \infty)}\]for each component of $z$.
If we assume $x_2$ is also bounded but not specified, then one cannot compute finite intervals.
Attacker's knowledge:
Attacker's capabilities:
Attacker's goals:
Label poisoning.
Feature poisoning.
Gradient-based poisoning.
Inner Problem: Learner's loss (Minimization)
Outer Problem: Attacker's objective (Maximization)
Definition: The attacker injects a hidden behavior into the model.
behavior:
Key Components:
By Defender's Goal
By DNN Lifecycle
In-training
Post-training
Consider a supervised learning model $f(x; \theta)$ trained using loss function $\mathcal{L}$. An attacker aims to perform a untargeted data poisoning attack by injecting a small set of poisoned samples $D_{poison} = {(x_p, y_p)}$ into the training dataset.
The attacker's goal is to cause a specific target input $x_t$ to be misclassified to any label $y_{adv} \neq y_t$.
Answer the following questions:
(i) Formulate the poisoning attack as a bilevel optimization problem. Clearly define the outer and inner problems.
(ii) Revise the optimization problem in (i) to ensure stealthiness by llimiting the deviation of poisoned samples from clean data.
Solutions:
(i) Bilevel optimization formulation for untargeted data poisoning
The attacker wants to craft a small set of poisoned samples $D_{\text{poison}} = {(x_p, y_p)}$ such that, after training on the combined dataset $D_{\text{clean}} \cup D_{\text{poison}}$, the resulting model misclassifies a specific target input $x_t$ to any label $y_{\text{adv}} \neq y_t$.
Inner problem (training): The defender trains the model by minimizing the loss on the poisoned training set:
\[\theta^*(D_{\text{poison}}) = \arg\min_{\theta} \sum_{(x,y) \in D_{\text{clean}} \cup D_{\text{poison}}} \mathcal{L}(f(x; \theta), y)\]Outer problem (attacker): The attacker chooses $D_{\text{poison}}$ to maximize the loss on the target input $x_t$ with incorrect label $y_{\text{adv}}$, or equivalently to cause misclassification:
\[\max_{D_{\text{poison}}} \mathcal{L}(f(x_t; \theta^*(D_{\text{poison}})), y_{\text{adv}})\]| subject to constraints such as $ | D_{\text{poison}} | \leq \epsilon$ (small fraction of total data). |
Thus the full bilevel problem is:
\[\begin{aligned} &\max_{D_{\text{poison}}} \mathcal{L}(f(x_t; \theta^{*}), y_{\text{adv}}) \\ &\text{s.t. } \theta^{*} = \arg\min_{\theta} \sum_{(x,y) \in D_{\text{clean}} \cup D_{\text{poison}}} \mathcal{L}(f(x; \theta), y), \\ &\quad |D_{\text{poison}}| \leq \epsilon, \quad y_{\text{adv}} \neq y_t \end{aligned}\](ii) Revised formulation with stealthiness constraint
To ensure stealthiness, we want the poisoned samples to look similar to clean data so they are hard to detect by data inspection or outlier detection. This is typically enforced by limiting the deviation of each poisoned sample $(x_p, y_p)$ from some clean base sample or from the clean data distribution.
One common approach constrains the $\ell_p$-norm difference between the poisoned features and their clean counterparts:
\[\begin{aligned} &\max_{D_{\text{poison}}} \mathcal{L}(f(x_t; \theta^{*}), y_{\text{adv}}) \\ &\text{s.t. } \theta^{*} = \arg\min_{\theta} \sum_{(x,y) \in D_{\text{clean}} \cup D_{\text{poison}}} \mathcal{L}(f(x; \theta), y), \\ &\quad |D_{\text{poison}}| \leq \epsilon, \quad y_{\text{adv}} \neq y_t, \\ &\quad {\vert x_p - x_{\text{base}}\vert}_p \leq \delta \quad \text{for each poisoned sample}, \\ &\quad y_p \text{ is plausible (e.g., correctly labeled for the base image)}. \end{aligned}\]The constraint ${\vert x_p - x_{\text{base}}\vert}_p \leq \delta$ ensures that poisoned samples are close to some clean data point $x{\text{base}}$ (e.g., the original clean version of that sample), making them visually or statistically similar to legitimate training data. The bound $\delta$ controls the stealthiness level—smaller $\delta$ means more stealthy but possibly less effective attack.
Membership inference:
Attribute inference:
Model inversion:
Training example extraction:
]]>⚠️ WARNING: THE COMMANDS/SCRIPTS IN THIS PASSAGE ARE ALMOST RUN UNDER
rootPRIVILEDGES. PLEASE BE CAREFUL AND MAKE SURE WHAT YOU ARE DOING BEFORE RUNNING THOSE COMMANDS/SCRIPTS IN REAL ENVIRONMENTS.⚠️ 警告：本文所包含之命令/腳本幾乎全部都使用
root權限執行。請務必小心並且確保你知道自己喺度做緊乜嘢之後再於真實環境之中執行命令。
ℹ Tips: The terms "master" and "slave" used in this article are only used to indicate the architectural relationship between database servers and do not refer to any group or have any derogatory connotation.
ℹ 提示：本文中所出現之「主」「從」等字眼只係用於表示數據庫服務器之架構關係，並無指代任何群體且無任何貶意。
This documentation is for the installation, deployment, upgrading, main/duplicated database, backup and restoration.
本文檔適用於 MySQL 喺 Linux 服務器環境之下嘅安裝部署、系統升級、主備數據庫搭建以及備份恢復之操作。
Under the Linux system environment, there are mainly 3 ways to install MySQL: install via rpm, install via binary packages and build from source code. Installing via rpm is simple, but it is not flexible enough, and one machine can only install one copy of MySQL database. The way building from source code is complex and may take a long time, but it can be configured by hand, so we do not take this way in general. This documentation recommends the second method to complete the installation.
喺 Linux 系統環境下， MySQL 嘅安裝主要有三種方式：通過 rpm 安裝、使用二進制軟件包安裝以及使用源碼編譯安裝。採用 rpm 包安裝嘅方式操作簡單，但其缺點係唔夠靈活，而且一台機器只可以安裝一個 MySQL 數據庫；採用二進制軟件包安裝嘅方式操作較為繁瑣且耗時較長，但可以根據實際情況手動優化編譯選項，一般情況下唔需要使用呢種方法進行安裝部署。本文檔描述採用 MySQL 二進制軟件包進行安裝部署之操作步驟。
For version upgrade operations in MySQL, this document stipulates that the number before the first decimal point from left to right in the post-MySQL version number is the large version number, and the number after the decimal point is the small version number. For example, in MySQL 8.0.36, 8 is the major version and 0.36 is the minor version; Small version upgrades adopt direct upgrade scheme, large version upgrade adopts logical data migration upgrade scheme. It should be noted that in the event of a MySQL version upgrade involving changes to configuration items in the MySQL configuration file and scrapping, the operation should be treated as a major version upgrade regardless of the version number size.
對於 MySQL 嘅版本升級操作，本文檔規定 MySQL 後版本數字中由左至右第一個小數點前嘅數字係大版本，小數點後嘅數字係小版本。例如 MySQL 8.0.36 中 8 係大版本，0.36 係小版本； MySQL 8.1 中 8 係大版本，1 係小版本。小版本升級採用直接升級方案，大版本升級採用邏輯數據遷移升級方案。需要注意嘅係，若涉及到 MySQL 配置文件中嘅配置項嘅更改及廢棄嘅 MySQL 版本升級，無論版本數字大小，都應該當做大版本升級嚟進行操作。
The MySQL main and duplicated database construction described in this document contains two architectures, a main-duplicated architecture (data is synchronized from the main to the duplicate) and a two-host architecture (two mains synchronize data with each other and are each other's duplicates). Building a main-duplicated library generally adopts a main-duplicated architecture, in which there must be a main node, the main library, and one or more duplicated nodes, the duplicated library. All data is written to the main node first, and then other nodes copy the incremental data on the main node, thus guaranteeing the consistency of the data. Building a main-duplicated database using a main-duplicated replication scheme will improve the availability and performance of the database. In a one-main-duplicated architecture, all data from the duplicated node (duplicated library) comes from the main node, and in general, the use of read-write separation model on this architecture is able to adequately resource, sending the client's read data request to the duplicated, the client's write request to the host, and then the duplicated synchronizes the newly written data from the main. In this case the throughput of the database can be greatly improved since the read-write requests are distributed to the duplicated and main processing respectively.
本文檔描述之 MySQL 主備數據庫搭建包含兩種架構，分別係一個主一從架構（數據從主機同步到從機）同兩個主機架構（兩台主機互相同步數據，互相係對方嘅從機）。搭建主備庫一般採用主從架構，其中必須有一個主節點，即主庫，以及一個或者多個嘅從節點，即備庫。所有數據都會先寫入到主節點，然後其他節點會複製主節點上嘅增量數據，從而保證數據嘅一致性。使用主從複製方案搭建主備庫將提升數據庫嘅可用性及性能。喺一主一從嘅架構中，從節點（備庫）嘅所有數據都係來自主節點，一般而言，喺呢個架構上使用讀寫分離嘅模式能夠充分資源，將客戶端嘅讀數據請求發送畀備機，將客戶端嘅寫數據請求發送畀主機，然後備機從主機處同步新寫入嘅數據。呢種情況下由於讀寫請求分別分攤到咗備機同主機上處理，所以可以極大提升數據庫嘅吞吐量。
Database backup derives and stores the data and structure of the database in an external location so that the database can be recovered in the event of data loss or corruption. There are two types of MySQL database backups described in this document: physical backup (raw backup) and logical backup. Among them, physical backups (logical backups) directly copy database files, including but not limited to data table files and log files, and logical backups backup SQL statement files, deriving all SQL statements that record data and structure operations.
數據庫備份將數據庫嘅數據同結構導出並儲存喺外部位置，令數據庫喺數據丟失或者損壞之時可以恢復。本文件描述嘅 MySQL 數據庫備份方式有物理備份（原始備份）同邏輯備份兩種。其中，物理備份（邏輯備份）直接複製數據庫文件，包括但唔限於數據表文件同日誌文件；邏輯備份則備份 SQL 語句文件，將記錄數據同結構操作嘅所有 SQL 語句導出。
Finally, this document contains some of the problems encountered during the testing process during MySQL installation, deployment, upgrade, backup, etc., and provides a problem analysis and solution for reference when actually deploying MySQL.
最後，本文件收錄咗一啲喺測試過程中喺 MySQL 安裝、部署、升級、備份等過程中遇到嘅問題，並提供咗問題分析及解決方法，以供實際部署 MySQL 時參考。
The text in this document using isowidth font with dark underprint is the command line raw text in the terminal. where the command that needs to be entered is preceded by a one-character permission prompt. The # symbol indicates that the next command was entered under the test machine's root privileges, the $ symbol indicates that the next command was entered under the test machine's ordinary user privileges, and the > symbol indicates that the next command requires a login to the MySQL command line console for operation. The permission prompt is automatically displayed by the computer terminal when the command is actually entered, eliminating the need for the user to manually enter this permission prompt as well as the space immediately following the symbol.
本文中使用等寬字體並帶有深色底紋嘅文本係終端中嘅命令行原始文本。其中需要輸入嘅命令前帶有一個字符嘅權限提示符。# 符號表示接下來嘅一條命令喺測試機嘅 root 權限下輸入；$ 符號表示接下來嘅一條命令喺測試機嘅普通用戶權限下輸入；> 符號表示接下來嘅命令需要登入 MySQL 命令行控制台進行操作。實際輸入命令時權限提示符號由電腦終端自動顯示，唔需要用戶手動輸入呢個權限提示符號以及符號後緊跟嘅空格。
Check the current system information about the computer, and make sure they are matched with the MySQL for installation. Use uname tool can check the current system and architecture information easily.
檢查當前機器嘅系統信息，確保呢啲信息同需要安裝嘅 MySQL 數據庫軟件相匹配。使用 uname 工具可以方便地查看當前嘅系統和架構信息。
# uname -a
Linux clz-db-t2 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Use ldd tool to check the version of system's GNU libc (Glibc).
使用 ldd 工具來檢查系統嘅 GNU libc (Glibc) 版本。
# ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
The compatibility of the current machine with the required MySQL version is determined based on the system version, architecture, and glibc version, thus deciding which MySQL version to use. For example, the test machine in this case runs Red Hat Enterprise Linux 7, an x86_64 architecture, and uses GNU libc 2.17. Therefore, when selecting the MySQL binary package, a version compatible with el7, an x86_64 architecture, and based on glibc 2.17 should be chosen.
兼容性要求當前呢台機器需要對應嘅 glibc 版本，因此亦都會決定邊啲 MySQL 版本可以使用。例如，依家情況下有一台測試機運行著 Red Hat Enterprise Linux，並且使用 GNU libc 2.17。因此，當選擇 MySQL 二進制包版本嘅時候就應該選擇一個兼容 el7，x86_64 架構，並且基於 glibc 2.17 嘅 MySQL 版本。
If the system architecture is displayed as aarch64, then an aarch64 or arm64 package should be selected; if the system architecture is displayed as x86_64, then an x86_64, x64, or amd64 package should be selected. The specific naming of the architecture used by the package depends on the software distributor.
如果系統架構顯示為 aarch64，咁應該選擇 aarch64 或者 arm64 包；如果系統架構顯示為 x86_64，咁應該選擇 x86_64、 x64 或者 amd64 包。包所用嘅架構嘅具體命名取決於軟件分銷商。
This document describes how to deploy MySQL using a binary package installation. From the official MySQL archive download page (https://downloads.mysql.com/archives/community/), select the required MySQL package and copy the link to that binary archive. On the machine where you want to deploy MySQL, use the wget tool to pull the MySQL binary package from that link and place it in the /usr/local directory. On the machine without an internet connection, use the ssh tool to upload the MySQL binary package from your local machine to the server.
呢份文檔會講解點樣用二進制包安裝部署 MySQL。喺官方嘅 MySQL 封存檔案下載頁面（https://downloads.mysql.com/archives/community/），選擇所需嘅 MySQL 包，然後將連結複製去嗰個二進制封存檔案。喺你想部署 MySQL 嘅機器上面，用 wget 工具喺嗰個連結度拉出 MySQL 二進制包，然後將佢放入 /usr/local 目錄。喺冇互聯網連線嘅機器上面，用 ssh 工具將 MySQL 二進制包由你嘅本地機器上載去伺服器。
# cd /usr/local
# wget "https://cdn.mysql.com/archives/mysql-8.0/mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz"
You can use the ls command in the /usr/local directory to view the files and directories in the current directory, ensuring that the MySQL binary package has been downloaded and placed in the specified location.
你可以用 /usr/local 目錄入面嘅 ls 命令嚟睇返當前目錄入面嘅檔案同目錄，確保 MySQL 二進制包已經下載好，然後放喺指定位置。
# ls -lah
total 428M
drwxr-xr-x. 16 root root 4.0K Jul 31 17:27 .
drwxr-xr-x. 14 root root 166 Feb 27 2019 ..
drwxr-xr-x. 2 root root 37 Aug 10 2020 bin
drwxr-xr-x. 2 root root 6 Mar 10 2016 etc
drwxr-xr-x. 2 root root 6 Mar 10 2016 games
drwxr-xr-x. 3 root root 21 Aug 10 2020 include
drwxr-xr-x. 2 root root 6 Mar 10 2016 lib
drwxr-xr-x. 4 root root 159 Aug 10 2020 lib64
drwxr-xr-x. 2 root root 6 Mar 10 2016 libexec
-rw-r--r-- 1 root root 428M Jun 28 2023 mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz
drwxr-xr-x 5 root root 44 Mar 26 2018 openssh.bak
drwxr-xr-x. 2 root root 6 Mar 10 2016 sbin
drwxr-xr-x. 6 root root 60 Aug 10 2020 share
drwxr-xr-x. 4 root root 106 Feb 27 2019 src
drwxr-xr-x 9 root root 188 Aug 10 2020 ssl
Use the tar tool to extract the package (in this example, "mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz") and place it in the same directory.
用「tar」工具去解壓縮個包（喺呢個例子入面，係「mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz」），然後將佢放入同一個目錄。
# tar -xvf mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz
The directory /usr/local/mysql-8.1.0-linux-glibc2.17-x86_64 is obtained.
得到咗個目錄「/usr/local/mysql-8.1.0-linux-glibc2.17-x86_64」。
Use the useradd command to create the mysql user group, and then create a new mysql user within that group.
用 useradd 命令去建立 mysql 用戶群組，然後喺嗰個群組入面建立一個新嘅 mysql 用戶。
# groupadd mysql
# useradd -r -g mysql -s /bin/false mysql
Configure the my.cnf configuration file as follows and save it in the /etc directory. You can open and paste it using the Vim editor.
配置 my.cnf 配置文件，然後將佢放入 /etc 目錄之中。你可以喺 Vim 編輯器之中打開並且貼上內容。
# vim /etc/my.cnf
Paste the following contents.
貼上下面呢啲內容。
[client]
#ssl-mode=REQUIRED
port = 3366
socket = /tmp/mysql.sock
[mysql]
prompt="\\u@\\h:\p \\R:\\m:\\s [\\d]>"
#tee=/data/mysql/mysql_3366/data/query.log
no-auto-rehash
default-character-set=utf8
socket = /tmp/mysql.sock
[mysqld]
#time_zone='+8:00'
require_secure_transport=off
basedir = /usr/local/mysql
#misc
user = mysql
datadir =/data/mysql/data/
port = 3366
socket = /tmp/mysql.sock
event_scheduler = off
#tmp
tmpdir=/tmp
#timeout
interactive_timeout = 1800
wait_timeout = 1800
#character set
character-set-server = utf8
lower_case_table_names=1
#character-set-server = utf8mb4
#sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
open_files_limit = 10240
max_connections =1000
max_connect_errors = 100
#monitor
#userstat=1
skip-name-resolve = on
#logs
log-output=file
slow_query_log = 1
slow_query_log_file = /data/mysql/logs/mysqlslow.log
log-error = /data/mysql/logs/mysqlerror.log
#log_error_verbosity =
pid-file = /tmp/mysql.pid
long_query_time = 0.1
#log-slow-admin-statements =
#log-queries-not-using-indexes =
log-slow-slave-statements = off
#binlog
binlog_checksum = CRC32
binlog_format = row
server-id = 22567
log-bin = /data/mysql/binlog/mysql-bin
log-bin-index = /data/mysql/binlog/mysql-bin.index
binlog_cache_size = 32M
max_binlog_size = 1G
max_binlog_cache_size = 1G
sync_binlog = 1
expire_logs_days = 7
sync_master_info=10000
#skip-log-bin
#relay log
relay-log = /data/mysql/binlog/mysql-relay-bin
skip_slave_start =on
max_relay_log_size = 1G
relay_log_purge =on
relay_log_recovery =on
log_slave_updates = on
#slave-skip-errors=,,
sync_relay_log_info=10000
slave_rows_search_algorithms=INDEX_SCAN,HASH_SCAN
explicit_defaults_for_timestamp=off
#buffers & cache
table_open_cache = 10000
table_definition_cache = 10000
max_allowed_packet = 1073741824
max_heap_table_size = 256M
sort_buffer_size = 16M
join_buffer_size = 16M
#thread_cache_size = 0
#query_cache_size = 0
#query_cache_type = 0
#query_cache_limit = 256K
thread_stack = 192K
tmp_table_size = 96M
key_buffer_size = 8M
read_buffer_size = 16M
read_rnd_buffer_size = 16M
bulk_insert_buffer_size = 32M
#myisam
#myisam_sort_buffer_size = 128M
#myisam_max_sort_file_size = 10G
#myisam_repair_threads =
#innodb
innodb_adaptive_hash_index=off
innodb_buffer_pool_size = 4G
innodb_buffer_pool_instances = 43
innodb_data_file_path = ibdata1:1G:autoextend
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 1G
innodb_log_file_size = 2G
innodb_log_files_in_group = 3
#innodb_max_dirty_pages_pct =
innodb_file_per_table = on
#innodb_rollback_on_timeout
#innodb_status_file =
#innodb_io_capacity = 50000
innodb_io_capacity=200
innodb_io_capacity_max=400
transaction_isolation = REPEATABLE-READ
#innodb_flush_method = O_DIRECT
gtid_mode = ON
enforce_gtid_consistency = ON
master_info_repository = TABLE
relay-log-info-repository = TABLE
innodb_undo_tablespaces=0
innodb_undo_log_truncate = on
innodb_lock_wait_timeout=10
# replication
#rpl_semi_sync_master_enabled = 0;
#rpl_semi_sync_master_timeout = 3600;
# Two-Master configure
#server-
#auto-increment-offset =
#auto-increment-increment =
#server-
#auto-increment-offset =
#auto-increment-increment =
slave_preserve_commit_order = on
slave_transaction_retries = 10
log_timestamps = system
#show_compatibility_56 = on
slave_parallel_workers = 4
slave_parallel_type = LOGICAL_CLOCK
interactive_timeout=3600
wait_timeout=3600
lock_wait_timeout=60
max_user_connections=800
#log_warnings=0
#log_error_verbosity=0
#optimizer_switch='index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=off,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,duplicateweedout=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on,derived_merge=on,favor_range_scan=off'
[mysqld_safe]
nice=-19
open-files-limit=10240
Save this configuration file after pasting. Create the MySQL-related directories to store the database's data files and logs.
貼上之後儲存呢個設定檔案。建立 MySQL 相關目錄嚟儲存資料庫嘅數據檔案同記錄。
# mkdir -p /data/mysql/data
# mkdir -p /data/mysql/binlog
# mkdir -p /data/mysql/logs
After creating the directory, switch the current directory to /data and grant ownership of the directory to the mysql user and group.
建立目錄之後，將當前目錄切換為 /data，並將目錄嘅擁有權授予 mysql 用戶同群組。
# cd /data
# chown -R mysql:mysql /data/mysql/
Switch to the directory where the MySQL package is located, /usr/local.
切入去 MySQL 包嘅目錄，即係 /usr/local。
# cd /usr/local
To facilitate subsequent deployment and use, create a symbolic link to the MySQL package: /usr/local/mysl.
為咗方便部署及使用，創建一個軟鏈接，連接到 MySQL 嘅軟件包 /usr/local/mysl。
# ln -s /usr/local/mysql-8.1.0-linux-glibc2.17-x86_64 /usr/local/mysql
Check if the symbolic link was created successfully.
檢查此軟鏈接係咪已經成功創建。
# ls -l | grep mysql
lrwxrwxrwx 1 root root 45 Jul 24 13:50 mysql -> /usr/local/mysql-8.1.0-linux-glibc2.17-x86_64
drwxr-xr-x 9 root root 129 Jul 24 13:40 mysql-8.1.0-linux-glibc2.17-x86_64
-rw-r--r-- 1 root root 448377800 Jun 28 2023 mysql-8.1.0-linux-glibc2.17-x86_64.tar.xz
Use the mysqld program to initialize the MySQL server.
使用 mysqld 程式來初始化 MySQL 伺服器。
# /usr/local/mysql/bin/mysqld --initialize --user=mysql
Copy the startup script that comes with the MySQL support files and place it in the /etc/init.d directory.
複製 MySQL 支援檔案隨附嘅啟動指令腳本，然後將佢放入 /etc/init.d 目錄。
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql.server
Use service tool to start the MySQL services.
使用 service 工具來啟動 MySQL 服務。
# service mysql.server start
When you see the message "Starting MySQL... SUCCESS!", it means the MySQL database has started successfully. The temporary password generated upon initial startup can be found in the MySQL logs. This password will be required for the first login, and you should change your account password immediately after logging into MySQL. According to the configuration file my.cnf, the MySQL error log is stored in /data/mysql/logs/mysqlerror.log. You can directly use the cat and grep tools to search for the keyword "password" to find the temporary password generated by MySQL.
當你見到「Starting MySQL... SUCCESS!」嘅訊息，即係 MySQL 資料庫已經成功啟動。初次啟動時產生嘅臨時密碼可以喺 MySQL 記錄入面搵到。第一次登入需要呢個密碼，而你應該喺登入 MySQL 之後即刻更改你嘅帳戶密碼。根據配置檔案 my.cnf， MySQL 錯誤記錄儲存喺 /data/mysql/logs/mysqlerror.log。你可以直接用 cat 同 grep 工具去搜尋關鍵字「password」嚟搵 MySQL 生成嘅臨時密碼。
# cat /data/mysql/logs/mysqlerror.log | grep password
2025-07-21T17:10:07.293097+08:00 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: jnPW*+Lse52Z
"jnPW*+Lse52Z" is the temporary password generated in this example. It's important to note that the temporary password initialized after each MySQL deployment is different; therefore, you must check the logs for this password after each deployment and initialization. Use this password to log in to the MySQL server.
「jnPW*+Lse52Z」係呢個例子入面生成嘅臨時密碼。重要嘅係要留意，每次 MySQL 部署之後初始化嘅臨時密碼都唔同；所以，你必須喺每次部署同初始化之後檢查呢個密碼嘅記錄。用呢個密碼登入 MySQL 伺服器。
# /usr/local/mysql/bin/mysql -uroot -p
Follow the prompts to enter your password. The appearance of the ">" sign indicates successful login to the MySQL command-line interface. Change the password for the root account.
跟住啲提示輸入你嘅密碼。出現「>」號表示成功登入 MySQL 命令行介面。更改 root 帳戶嘅密碼。
> ALTER USER 'root'@'localhost' IDENTIFIED BY 'newpassword';
Using the MySQL root user for application connections is insecure; therefore, a dedicated account should be created for the application. Following the principle of least privilege, the application-specific account should be granted the necessary permissions. The application account name should be related to the current database ID or the application itself to avoid confusion. For example, on a server with server-id 13, create an account for the application, name it "apn_13", and grant it read and write permissions. The CREATE, DELETE, and DROP permissions should be granted cautiously based on the specific application requirements.
用 MySQL 根用戶嚟連接應用程式係唔安全嘅；所以，應該為個應用程式建立一個專用嘅帳戶。根據最低特權原則，應用程式專用帳戶應該獲得必要嘅權限。應用程式帳戶名稱應該同當前嘅資料庫 ID 或者應用程式本身有關，以免混淆。例如，喺伺服器 ID 為 13 嘅伺服器上面，為應用程式建立一個帳戶，將佢命名為「apn_13」，然後授予佢讀寫權限。應該根據特定嘅應用程式要求，謹慎噉授予 CREATE、DELETE 同 DROP 權限。
> CREATE USER 'apn_13'@'localhost' IDENTIFIED BY 'newpassword';
> GRANT SELECT, INSERT, UPDATE ON database_name.database_table TO 'apn_13'@'localhost';
In the command to create a user, the "newpassword" field needs to be set to a new password that meets the security requirements and is different from the root user's password. In the command to grant permissions, the "database_name" and "database_table" fields should be filled in according to the application's usage. If the application uses all databases and tables in the database, you can fill in "*.*" to represent all databases and tables, for example:
喺建立用戶嘅命令入面，「newpassword」欄位需要設定為一個符合安全要求嘅新密碼，同埋同 root 用戶嘅密碼唔同。喺授予權限嘅命令入面，應該根據應用程式嘅用途填寫「database_name」同「database_table」欄位。如果應用程式用咗資料庫入面嘅所有資料庫同表，你可以填「*.*」嚟表示所有資料庫同表，例如：
> GRANT SELECT, INSERT, UPDATE ON *.* TO 'apn_13'@'localhost';
Check the current system environment. We can query for the system information and its MySQL version information from terminal commands. For example the test machine T2:
- IP: 10.91.134.xxx
- OS: Red Hat Enterprise Linux 7
- CPU Architecture: x86_64
- MySQL: 8.0.36 MySQL Community Server
- GLIBC: GNU libc 2.17
From the information above, we can confirmed that the running MySQL version is MySQL 8.0.36.
Then we confirm the service status.
# service mysql.service status
Log into MySQL command line interface and query for current MySQL information in service.
# /usr/local/mysql/bin/mysql -u root -p
Check the current version information.
> SELECT Version();
And then get the version.
+-----------+
| version() |
+-----------+
| 8.0.36 |
+-----------+
1 row in set (0.00 sec)
Copy download link from official MySQL website and pull down to machine via wget tool. Put the newly downloaded package under the directory /usr/local. In this example we need to upgrade MySQL 8.0.36 to MySQL 8.0.42 so we just wget the package of MySQL 8.0.42.
# cd /usr/local
# wget "https://downloads.mysql.com/archives/get/p/23/file/mysql-8.0.42-linux-glibc2.28-x86_64.tar.xz"
Wait until the progress bar finished, then check the /usr/local directory to see whether the package mysql-8.0.42-linux-glibc2.28-x86_64.tar.xz exists. Then use tar to extract the package.
# tar -xvf mysql-8.0.42-linux-glibc2.17-x86_64.tar.xz
Get the directory for new version.
Release the connection to MySQL database to prevent the read/write problem. This step should be done according to the actual situation.
Log into MySQL and run dirty-data writing operations.
> SET GLOBAL innodb_fast_shutdown = 0;
Log out MySQL and shut down MySQL services.
# /usr/local/mysql/bin/mysqladmin -u root -p shutdown
The services will be closed after the password is entered. Check the launch daemons to make sure the services are stopped.
# service mysqld status
If "ERROR! MySQL is not running, but lock file (/var/lock/subsys/mysql) exists" shows up, the MySQL services are shutted down correctly.
Remove the soft link of old version MySQL, and create a new soft link for new version.
# rm -f /usr/local/usr
# ln -s /usr/local/mysql-8.0.42-linux-glibc2.28-x86_64 /usr/local/mysql
Use mysqld_safe launch the new version MySQL temporary, then use mysql_upgrade tool attached to MySQL to check the compatibility.
# /usr/local/mysql/bin/mysql_safe --user=mysql
# /usr/local/mysql/bin/mysql_upgrade -u root -p
Then we will get the hints.
The mysql_upgrade client is now deprecated. The actions executed by the upgrade client are now done by the server.
To upgrade, please start the new MySQL binary with the older data directory. Repairing user tables is done automatically. Restart is not required after upgrade.
The upgrade process automatically starts on running a new MySQL binary with an older data directory. To avoid accidental upgrades, please use the --upgrade=NONE option with the MySQL binary. The option --upgrade=FORCE is also provided to run the server upgrade sequence on demand.
It may be possible that the server upgrade fails due to a number of reasons. In that case, the upgrade sequence will run again during the next MySQL server start. If the server upgrade fails repeatedly, the server can be started with the --upgrade=MINIMAL option to start the server without executing the upgrade sequence, thus allowing users to manually rectify the problem.
If there is no compatibility problem we can restart the database. Copy one new version launch daemon from support-files directory to /etc/init.d.
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
# service mysqld start
Then log into the MySQL and verify the current version.
> SELECT Version();
If we run into any compatibility related problems in previous steps, we need to roll back. Restore the MySQL soft link to previous status and restart. Stop MySQL service first if there is MySQL service running.
# service mysqld stop
or
# /usr/local/mysql/support-files/mysql.server stop
remove the current version soft link.
# rm -f /usr/local/mysql
and create soft link of previous version.
# ln -s /usr/local/mysql-8.0.36-linux-glibc2.28-x86_64 /usr/local/mysql
start previous version's services.
# /usr/local/mysql/support-files/mysql.server start
Log into MySQL to check the version is rolled back to the previous status. If everything is fine, remove the launch daemon in /etc/init.d, and copy one prevous launch daemon to the /etc/init.d directory.
# rm -f /etc/init.d/mysqld
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
Confirm the current environment of the machine. From the terminal we can get the current system and MySQL information. In this example, they are:
- IP: 10.91.134.136
- OS: Red Hat Enterprise Linux 7
- CPU Architecture: x86_64
- MySQL: 8.0.36 MySQL Community Server
- GLIBC: GNU libc 2.17
The currently running MySQL is version 8.0.36, and we can easily check the status of MySQL services.
# service mysqld status
Log into MySQL and query for the current MySQL version which is serving.
# /usr/local/mysql/bin/mysql -u root -p
Confirm the current version.
> SELECT Version();
We can get the version is 8.0.36.
+-----------+
| version() |
+-----------+
| 8.0.36 |
+-----------+
1 row in set (0.00 sec)
Download MySQL from the official MySQL archive website. Copy the target link and use wget tool to download package to machine. Put the newly downloaded MySQL package in /usr/local directory. In this example case, we need to upgrade MySQL from 8.0.42 to MySQL 9.3.0, so we wget the MySQL 9.3.0 package directly.
# cd /usr/local
# wget "https://downloads.mysql.com/archives/get/p/23/file/mysql-9.3.0-linux-glibc2.17-x86_64.tar.xz"
Wait for the progress bar and check the /usr/local directory to see whether there is a mysql-9.3.0-linux-glibc2.17-x86_64.tar.xz package. Then extract the package via tar tool.
# tar -xvf mysql-9.3.0-linux-glibc2.17-x86_64.tar.xz
Now we get the new version MySQL directory. In this case it is mysql-9.3.0-linux-glibc2.17-x86_64.
# /usr/local/mysql/bin/mysqldump -u root -p --add-drop-table --routines --events --all-databases --force > backup_file.sql
The exported data is contained in the sql file.
Release the connection to database.
Run the write dirty data operation. This operation may need a longer time, and it depends on the amount of dirty data and the speed of the disks.
> SET GLOBAL innodb_fast_shutdown = 0;
Remove old MySQL soft link first, and then create a new MySQL soft link for new version.
# rm -f /usr/local/mysql
# ln -s /usr/local/mysql-9.3.0-linux-glibc2.28-x86_64 /usr/local/mysql
Backup old MySQL configurations my.cnf and move the file to a backup directory.
# mv /etc/my.cnf /data/backup/backup_8.0.42.cnf
Check the configuration file for new MySQL to make sure it is matched with the update descriptions of newer MySQL versions. For those changed configurations items and deprecated items, new alternatives should be applied to the configuration file. For issues made by changes of the configurations, see details in Chapter Diagnose. Put the new version's configuration file in /etc directory and name it as my.cnf.
Change to /data directory. Backup old MySQL related directories as mysql_bak.
# cd /data
# mv mysql mysql_bak
Create new MySQL directory under /data directory.
# mkdir -p /data/mysql/data
# mkdir -p /data/mysql/binlog
# mkdir -p /data/mysql/logs
And its structure should be like this.
mysql
├── binlog
├── data
└── logs
Change the owner and privileges of the directory and subdirectories to mysql:mysql.
# chown -R mysql:mysql /data/mysql
Use ls -lah command to see detailed privilege information and make sure the change is successful.
# ls -lah | grep mysql
drwxr-xr-x 5 mysql mysql 44 Jul 22 15:33 mysql
Initialize the MySQL database.
# /usr/local/mysql/bin/mysql --initialize --user=mysql
If the initialization is successful, we can start the database. Copy one launch daemon to /etc/init.d first.
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysqld
# service mysqld start
When we finish the construction of the database and start services, we may need to migrate the original data from the previous database to new database. In this part we describe the operations that imports previous data to new database.
Create new database. According to the data scripts (usually sql files), setup databases with same names as previous databases. Generally, the names of the databases should be same of names contained in exported sql scripts. Additionally, exported backup scripts also contain exporting time and other information.
> CREATE DATABASE original_data;
Change into the newly created database original_data after the creation, and the original_data can be any name according to the actual situation.
> USE original_data;
Then upload the sql import script to the server. Log into the MySQL command line interface, import the data in original_data database. The position where the script is located should be an absolute path.
> SOURCE /path/to/original_data_backup_time.sql;
Wait for the importing process to finish. If there are several sql backup scripts need to import, we should import them by time order, from early to the latest.
To set up a MySQL master-slave database setup, at least one device is needed for the master database and one device for the slave database. Both devices should have the same MySQL database software version installed. In this example, both test machines are running MySQL version 8.1.0. The system environments of test machines t1 and t2 are as follows. For simplicity, the two machines used to set up the master-slave database setup will be referred to as t1 and t2.
Test Machine t1:
Linux clz-db-t1 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Test Machine t2:
Linux clz-db-t2 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
The two test machine have the same configuration and system environments, and they both have MySQL 8.1.0 database server installed. Start these two test machine, login and verify their MySQL version.
mysql.server servicesManaging the MySQL service directly through mysql.server: When not using the service utility to manage mysqld, starting the MySQL service requires using /mysqld/support-files/mysql.server. This method is suitable for MySQL software that was not installed using RPM.
Start MySQL services:
# /usr/local/mysql/support-files/mysql.server start
Stop MySQL services:
# /usr/local/mysql/support-files/mysql.server stop
Check MySQL status:
# /usr/local/mysql/support-files/mysql.server status
Managing MySQL services using systemctl: If the MySQL database software was installed via RPM, you can use systemctl to manage the MySQL service.
Enable MySQL services:
# systemctl enable mysqld.service
Start MySQL services:
# systemctl start mysqld.service
Check MySQL status:
# systemctl status mysqld.service
Stop MySQL services:
# systemctl stop mysqld.service
Disable MySQL services:
# systemctl disable mysqld.service
Restart MySQL services:
# systemctl restart mysqld.service
Managing the MySQL service using a service: For easier testing and management, you can use the mysqld service to control the MySQL service. Copy the mysql.server script from the support-files directory under your MySQL software installation directory to the /etc/init.d directory and rename it to mysqld. This method is suitable for MySQL server software installed using binary packages.
# cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql.server
Perform this process on both machines, and then you can use the service command to manage the MySQL service status.
Stop MySQL services:
# service mysql.server stop
Check MySQL status:
# service mysql.server status
Start MySQL services:
# service mysql.server start
Check the machine t1 that hosts the MySQL master database, log in and open the MySQL configuration file my.cnf. This configuration file should be located in the /etc directory.
# vim /etc/my.cnf
Check that the configuration items bin-log and server-id in the configuration file are set and not empty. The server-id can be set according to the actual situation; the test machine's server-id is set to 22567. Note that the server-id of the MySQL database master server and the database standby (slave) server must not be the same; otherwise, the database backup machine will be unable to connect to the master.
Create a backup account for master-slave replication on the MySQL master database server. First, list all accounts currently in MySQL for verification and comparison.
> SELECT user FROM mysql.user;
Then create a new backup user dedicated to master-slave replication, named replica_user.
> CREATE USER 'replica_user'@'%' IDENTIFIED BY 'newpassword';
Listing all users in MySQL again reveals that the newly created user, replica_user, has been added to the MySQL database.
> SELECT user FROM mysql.user;
+------------------+
| user |
+------------------+
| replica_user |
| mysql.infoschema |
| mysql.session |
| mysql.sys |
| root |
+------------------+
5 rows in set (0.00 sec)
Check the host settings to ensure that the log_bin binary log is enabled, and check the current host's master status. Find the status of the log_bin variable in the configuration; this status should be ON. If this status is OFF, modify this value in the configuration file my.cnf and restart the MySQL service.
> SHOW VARIABLES LIKE 'log_bin';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin | ON |
+---------------+-------+
1 row in set (0.01 sec)
Check the current MySQL host status. Use the following command in MySQL 8.0.
> SHOW MASTER STATUS \G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 157
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
Use the following command in MySQL version 8.4.
> SHOW BINARY LOG STATUS \G
*************************** 1. row ***************************
File: mysql-bin.000002
Position: 798
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: b6a6daa5-71a5-11f0-91cb-005056994dcc:1-2
1 row in set (0.00 sec)
Grant replica_user master-slave replication permissions.
> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replica_user'@'%';
Check that the configuration items bin-log and server-id in my.cnf are enabled and their values are not empty. The server-id of the MySQL backup machine in this test is set to 25566. In actual deployment, the setting can be adjusted according to the situation; generally, server-id is set to the last part of the server IP address. It is important to note that the server-id of the MySQL database host and the database backup machine (slave) must not be the same; otherwise, the database backup machine will be unable to connect to the host. If the slave node is only used as a database backup and there is no intention to switch it to the master, then bin-log logging is not required; otherwise, bin-log logging should be enabled for both the master and slave. Log in to the MySQL server on the slave machine. First, stop all master-slave replication operations. For MySQL 8.0 and above, use the following statement.
> STOP REPLICA;
For versions of MySQL below 8.0, use the following statement.
> STOP SLAVE;
Then, configure the MySQL master database address and account settings on the standby MySQL database. Use the following command in MySQL 8.1. The arrow "->" indicates a newline prompt; you don't need to copy it while typing, as it will appear automatically when you press Enter.
> CHANGE MASTER TO master_host='10.91.134.135',
-> master_port=3366,
-> master_user='replica_user',
-> master_password='newpassword',
-> master_log_file='mysql-bin.000001',
-> master_log_pos=157,
-> get_master_public_key=1;
In this configuration, master_host should be set to the IP address of the machine hosting the primary database, and the secondary database should be able to access this address. master_port should be set to the port used by the primary MySQL database. master_user should be set to the username created in the previous section for master-slave replication; in this example, it's replica_user. master_password should be set to the password for the master-slave replication account. master_log_file and master_log_pos should be filled in according to the host's status output. In MySQL 8.0 and later versions, the get_master_public_key option needs to be enabled; see the Troubleshooting section for details. After completion, restart master-slave replication. For MySQL 8.4 and later versions, the following command should be used.
> CHANGE REPLICATION SOURCE TO
-> source_host='10.91.134.135',
-> source_port=3366,
-> source_user='replica_user',
-> source_password='newpassword',
-> source_log_file='mysql-bin.000001',
-> source_log_pos=1046,
-> get_source_public_key=1;
In MySQL 8.0 and above:
> START REPLICA;
Or in versions of MySQL below 8.0:
> START SLAVE;
Check the primary/standby connection status by entering the following information on the standby machine.
> SHOW REPLICA STATUS \G
*************************** 1. row ***************************
Replica_IO_State: Waiting for source to send event
Source_Host: 10.91.134.135
Source_User: replica_user
Source_Port: 3366
Connect_Retry: 60
Source_Log_File: mysql-bin.000001
Read_Source_Log_Pos: 351
Relay_Log_File: mysql-relay-bin.000002
Relay_Log_Pos: 520
Relay_Source_Log_File: mysql-bin.000001
Replica_IO_Running: Yes
Replica_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Source_Log_Pos: 351
Relay_Log_Space: 730
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Source_SSL_Allowed: No
Source_SSL_CA_File:
Source_SSL_CA_Path:
Source_SSL_Cert:
Source_SSL_Cipher:
Source_SSL_Key:
Seconds_Behind_Source: 0
Source_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Source_Server_Id: 22567
Source_UUID: 4f6e1086-6852-11f0-aa5f-005056994dcc
Source_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
Source_Retry_Count: 10
Source_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Source_SSL_Crl:
Source_SSL_Crlpath:
Retrieved_Gtid_Set: 4f6e1086-6852-11f0-aa5f-005056994dcc:1
Executed_Gtid_Set: 4f6e1086-6852-11f0-aa5f-005056994dcc:1,
87ce2877-692d-11f0-8354-0050569982d7:1-3
Auto_Position: 0
Replicate_Rewrite_DB:
Channel_Name:
Source_TLS_Version:
Source_public_key_path:
Get_Source_public_key: 1
Network_Namespace:
1 row in set (0.00 sec)
If both Replica_IO_Running and Replica_SQL_Running show as Yes, it means that MySQL master-slave replication is running normally.
Check if the databases in the current primary and standby databases are consistent, and view the current database.
> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.01 sec)
Create a new test database in the main database, named dbtest1 in this example.
> CREATE DATABASE dbtest1;
Check if the database is included in the primary database.
> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| dbtest1 |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
Check if the backup database is synchronized.
> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| dbtest1 |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
It can be confirmed that the newly created database has been replicated from the primary database to the standby database. After confirming that the primary-standby synchronization and replication are normal, remove the test database and tables.
To achieve a master-slave architecture where the two MySQL servers act as each other's master and backup, and operations on either MySQL server are synchronized across both servers, master-slave replication needs to be configured on both MySQL servers.
For use in MySQL 8.4 and above:
> SHOW BINARY LOG STATUS \G
Used in MySQL versions 8.1 and below:
> SHOW MASTER STATUS \G
The following information can be obtained.
*************************** 1. row ***************************
File: mysql-bin.000003
Position: 4006
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: b6a6daa5-71a5-11f0-91cb-005056994dcc:8-18,
e3cbf8a0-71a6-11f0-8381-0050569982d7:1-4
1 row in set (0.00 sec)
Create a new account on t2 specifically for master-slave replication on t1, following the same procedure as creating a replication-dedicated account on t1. Log in to the MySQL console on t2.
> CREATE USER 'replica_user_2'@'%' IDENTIFIED BY 'newpassword';
Then grant the new user replica_user_2 the permissions related to master-slave replication.
> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replica_user_2'@'%';
Then log in to MySQL on t1 and first stop master-slave replication on t1.
> STOP REPLICA;
Then, configure the information for t2 in t1. Enter the IP address of t2 in source_host; enter the MySQL service port of t2 in source_port; enter the username created on t2 for master-slave replication (in this example, replica_user_2); enter the password corresponding to source_user; enter the name of the corresponding binary log file on t2 (in this example, mysql-bin.000003); enter the position of the binary log (e.g., 3012 in this example); finally, set get_source_public_key to 1 to retrieve the public key from the host.
> CHANGE REPLICATION SOURCE TO
-> source_host='10.91.134.136',
-> source_port=3366,
-> source_user='replica_user_2',
-> source_password='newpassword',
-> source_log_file='mysql-bin.000003',
-> source_log_pos=3012,
-> get_source_public_key=1;
Once completed, start master-slave replication on t1.
> START REPLICA;
Check the master-slave replication status of t1.
> SHOW REPLICA STATUS \G
...
Replica_IO_Running: Yes
Replica_SQL_Running: Yes
...
Master-slave replication is working normally. When a write operation is performed on machine t2, machine t1 can synchronize and modify the data normally.
Physical backups consist of original copies of the directories and files storing the database content. This type of backup is suitable for large or critical databases that require rapid recovery in the event of problems. Physical backups are faster than logical backups because they involve only file copying operations and do not require database transformation. In addition to the database itself, physical backups can include any related files such as logs or configuration files. Physical backup files can only be migrated to other machines with the same or similar hardware characteristics. The granularity of physical backups ranges from the entire data directory level to the individual file level, but may not necessarily offer table-level granularity, depending on the storage engine used.
Perform backups while the database is completely shut down. The advantages of this method are high data consistency and fast backup and recovery speeds. However, the disadvantage is that it requires system downtime, which can significantly impact business operations. Alternatively, if conditions allow for a cold backup, disconnect the application from the database, stop reading and writing data, and shut down the MySQL database.
# service mysqld stop
A properly functioning MySQL database stores data in the /data/mysql directory and its configuration file in /etc/my.cnf. Therefore, a cold backup only needs to back up these two directories. Create a new directory /backup/offline under /data to store the backup files for the cold backup.
# mkdir -p /data/backup/offline
Copy the entire /data/mysql directory to the /data/backup/offline directory, and rename it, adding the backup time. For example:
# cp -r /data/mysql /data/backup/offline/backup_data_2025_08_04
Next, copy the entire my.cnf configuration file to the mysql_backup_offline directory, rename it, and add the backup time. For example:
# cp /etc/my.cnf /data/backup/offline/backup_config_2025_08_04.cnf
For specific, recurring database backup operations, a script named mysql_backup_offline.sh can be created to perform a cold backup of the entire database. The shell script is as follows.
backup_date=$(date "+%Y_%m_%d")
backup_folder=/data/backup/offline
data_dir_name="backup_data_${backup_date}"
config_name="backup_config_${backup_date}"
echo "[ INFO ] Starting offline backup process..."
echo "[ INFO ] Stoping MySQL service..."
service mysqld stop
service mysqld status
echo "[ INFO ] MySQL service stopped."
echo "[ INFO ] Backup databases..."
cp -r /data/mysql ${backup_folder}/backup_data_${backup_date}
tar -zcvPf ${backup_folder}/backup_data_${backup_date}.tar.gz ${backup_folder}/backup_data_${backup_date}
echo "[ INFO ] Backup configuration..."
cp /etc/my.cnf ${backup_folder}/backup_data_${backup_date}
echo "[ INFO ] Finished."
When a cold backup of the database is required, the script will automatically perform a cold backup of the database and name the backup file.
# cd /usr/local
# sh mysql_backup_offline.sh
Restart the MySQL service after the cold backup is complete.
# service mysqld start
Performing backups while the database is running will not affect its normal operation. However, using a hot backup strategy on a MySQL database requires additional backup tools, such as the open-source database backup tool Percona XtraBackup. Use wget to pull the corresponding version of the XtraBackup binary package (rpm format) from the XtraBackup website and place it in the /usr/local directory.
# cd /usr/local
# wget "https://downloads.percona.com/downloads/Percona-XtraBackup-8.4/Percona-XtraBackup-8.4.0-1/binary/redhat/7/x86_64/percona-xtrabackup-84-8.4.0-1.1.el7.x86_64.rpm"
Install the XtraBackup package.
# yum localinstall percona-xtrabackup-80-8.0.32-26.1.el7.x86_64.rpm
If you are prompted that libev.so.4 is missing, you need to install libev, a dependency required by XtraBackup.
# yum localinstall libev-4.04-2.el6.x86_64.rpm
If you are prompted that openssl is missing, you need to install the OpenSSL dependency required by XtraBackup.
# yum install openssl
If you are prompted that zstd is missing, you need to install Zstandard (usually abbreviated as zstd), a dependency required by XtraBackup. If zstd is available in the package manager's repository, you can install it directly through the package manager.
# yum install zstd
If the zstd package is not included in the yum repository, you can pull the source package from the zstd project website and compile it for installation, or pull the RPM package from another reliable mirror server to your local machine for installation. Place the zstd RPM package in the /usr/local directory.
# rpm -ivh zstd-1.4.0-1.el6.x86_64.rpm
Alternatively, you can use yum to install locally.
# yum localinstall zstd-1.4.0-1.el6.x86_64.rpm
After installing XtraBackup, use the command to verify that the version of the current backup tool matches the version of the running MySQL service.
# xtrabackup --version
xtrabackup version 8.4.0-1 based on MySQL server 8.4.0 Linux (x86_64) (revision id: da6e1abd)
For databases that require regular hot backups, the following script can be used to simplify the commands entered each time, automatically name the backup files according to a predetermined method, and avoid backup file name conflicts.
echo "[ INFO ] Starting XtraBackup..."
current_date=$(date "+%Y_%m_%d")
backup_dir=/data/backup/xtrabackup
xtrabackup --backup --slave-info -u root -H localhost -P3366 -p --target-dir=${backup_dir}/backup_${current_date} 2> ${backup_dir}/backup_${current_date}.log
echo "[ INFO ] Compressing..."
tar -zcvPf ${backup_dir}/backup_${current_date}.tar.gz ${backup_dir}/backup_${current_date}/
echo "[ INFO ] XtraBackup done."
The mysqldump tool can be used to back up an entire database or a single table. For a MySQL server installed in the /usr/local/mysql directory, use the following command to start the mysqldump backup.
# /usr/local/mysql/bin/mysqldump -u root -p --lock-all-tables --flush-logs --set-gtid-purged=off database_name > backup_name.sql
Set the backup location for the MySQL database to the dump folder under the /data/backup directory. Name the backup file as follows: dump_database_name_yyyymmddHHMM.sql, for example, dump_dbtest1_202507311541.sql indicates that this backup is for the database dbtest1, created on July 31, 2025. Include the backup creation date in the filename to prevent duplicate backup file names. For regular logical backups of a specific database, the following script can simplify the commands entered each time.
current_date=$(date "+%Y%m%d%H%M")
backup_folder=/data/backup/dump
username="root"
db_name="db1"
file_name="dump_${db_name}_${current_date}.sql"
backup_dir_file_name="${backup_folder}/${file_name}"
echo "[ INFO ] Starting backup MySQL database ${db_name} at ${current_date}..."
/usr/local/mysql/bin/mysqldump -u${username} -p --lock-all-tables --flush-logs --set-gtid-purged=off ${db_name} > ${backup_dir_file_name}
cd ${backup_folder}
tar -zcvPf ${file_name}.tar.gz ${file_name}
finish_date=$(date "+%Y%m%d%H%M")
echo "[ INFO ] Finish backup MySQL database ${db_name} at ${finish_date}."
つづく…
]]>⚠️ THIS POST HAS A "靈車漂移" TAG, AND DOES NOT GUARANTEE STABILITY.
⚠️ 呢篇文章帶有「靈車漂移」之標籤，並且不保證穩定性。
Recently Rain is using a combination of "separated" NAS and remote desktop, that is, an Intel NUC with a mounted external hard disk. It is worth attention that I want to have a user-friendly command line environment as well as a beautiful graphic user interface (GUI), so I installed a macOS Big Sur 11.9 on NUC, also called "Hacintosh". I will called it "Nucintosh" because it is installed on a NUC hardware.
呢排 Rain 用緊一個「散裝」NAS 及遠程桌面結合物，即係一台 Intel NUC，並掛載有一個外置硬碟。值得注意嘅係 Rain 想要一個用戶友好嘅命令行終端環境，又同時想要一個優雅美麗嘅圖形用戶介面，所以選擇喺 NUC 之上安裝 macOS Big Sur 11.9 系統，即係話安裝一個「黑蘋果」。因為佢運行於 NUC 硬件之上，所以喺下文中將稱呼佢為「Nucintosh」。
It is generally acknowledge that macOS use ARD to connect from remote, and it does not support the RDP protocol form Microsoft. Rain wants to make Nucintosh be connected via RDP, so that maybe I can adjust the resolution settings from remote client. (not sure)
一般來講，macOS 係用 ARD 喺遠端連線，而且佢唔支援微軟嘅 RDP 協定。 Rain 想令到 Nucintosh 透過 RDP 連接，咁我可能可以喺遠端客戶端調整解像度嘅設定。（唔確定）
The method is to build an RDP server on macOS and make it run rdp services, and there is an open source solution called xrdp (GitHub repo) but however it does not support macOS officially.
此方法係專門構建一個 RDP 伺服器於 macOS 系統之上，令到佢運行 RDP 服務。而且有一個叫做「xrdp」嘅開源嘅解決方案，但係佢冇官方嘅 macOS 支援。
According to some discussions, there is a way to make Macs run xrdp and get remote connected.
根據一啲討論，有一種方法可以令到 Mac 電腦運行 xrdp 並使得可以被遠程連接控制。
I choose xrdp 0.9.19 version for this task. Someone said this version runs well on macOS Monterey. My Nucintosh runs a macOS Big Sur system, which is earlier than macOS Monterey, so it has a large probability to work with xrdp 0.9.19 as well. Let us have a try.
我選擇 xrdp 0.9.19 呢個版本。有啲人話呢個版本喺 macOS Monterey 上面運行良好。我嘅 Nucintosh 搭載一個 macOS Big Sur 系統，相比於 Monterey 早一個大版本，因此有相當大可能性令到 xrdp 0.9.19 運行良好。讓我哋試一試。
We need these following dependencies:
我哋需要下面呢啲依賴庫：
If the machine do not have Homebrew installed, it is required to install one.
如果機器上面冇安裝 Homebrew，咁應該安裝一個先。
First, get Xcode command line tools ready.
首先，準備好 Xcode 命令行工具（Xcode Commandline Tools）。
xcode-select --install
Then install Homebrew.
然後安裝 Homebrew。
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
It is also okay to have the macOS has MacPorts installed, like my Nucintosh, but it is also highly recommend to install Homebrew in this case. Some dependencies seem can only be installed from Homebrew, like XQuartz components.
如果 macOS 已經安裝咗 MacPorts 都冇問題。好似 Nucintosh 一樣既有 MacPorts 又有 Homebrew。但係仍然非常推薦安裝 Homebrew，因為我哋後面需要用到。一啲依賴庫只能夠從 Homebrew 嗰度安裝，例如 XQuartz 嘅組件。
Install openssh.
安裝 openssh。
brew install openssh
Install automake and autoconfig.
安裝 automake 同埋 autoconfig。
brew install automake
Install libtool.
安裝 libtool。
brew install libtool
Install pkgconfig.
安裝 pkgconfig。
brew install pkgconfig
Install nasm.
安裝 nasm。
brew install nasm
Install XQuartz and its related libraries.
安裝 XQuartz 及其相關嘅庫。
brew install xquartz
brew install libx11
brew install libxfixes
brew install libxrandr
Make sure all the dependencies above are installed.
確保所有嘅依賴都已經安裝好。
All the operations are completed via terminal and CLI, so the only thing now I need to do is to download xrdp 0.9.19 source code archive via command line. I use wget to do this.
所有嘅操作都係喺終端嘅命令行介面完成嘅，因此依家需要通過命令行工具來下載 xrdp 0.9.19 源代碼包。使用 wget 即可做到。
wget "https://github.com/neutrinolabs/xrdp/releases/download/v0.9.19/xrdp-0.9.19.tar.gz"
If the system do not have a wget tool installed, we can install from Homebrew easily. (if neccessary)
如果系統上面冇安裝 wget 工具，可以使用 Homebrew 來安裝。（如果必要嘅話）
brew install wget
Move this tar file to a coding place, for example, I put it in ~/Coding directory.
將呢個 tar 歸檔文件移動去一個編寫代碼嘅常用位置，例如我將佢放喺 ~/Coding 嘅目錄下面。
mv xrdp-0.9.19.tar.gz ~/Coding
Use tar tool to extract the package.
使用 tar 工具解壓源碼包。
tar -xvf xrdp-0.9.19.tar.gz
Then cd into the directory.
然後 cd 入嗰個目錄。
cd xrdp-0.9.19
These steps are for x86_64 Macs.
呢啲步驟適用於 x86_64 架構嘅 Mac 電腦。
Start.
開始配置。
./bootstrap
Configure the project. If the system does not have pkgconfig installed, it will throw an error out here.
配置項目。如果個系統冇安裝有 pkgconfig，佢會丟出一個報錯喺度。
./configure PKG_CONFIG_PATH=/opt/X11/lib/pkgconfig
Make and build.
構建。
make
Install xrdp.
安裝 xrdp。
sudo make install
The xrdp will be installed at /usr/local/sbin/xrdp by default.
xrdp 呢個程序將會默認被安裝喺 /usr/local/sbin/xrdp 嘅位置。
Make sure xrdp is installed on macOS.
確保 xrdp 已經安裝喺 macOS 之中。
which xrdp
If the output is /usr/local/sbin/xrdp, then xrdp is installed correctly. Generally, if no changes were made in source code before installation, the configuration file xrdp.ini will be stored at /etc/rxdp. Use vim or other text editor to open it, with root privilige.
如果輸出係 /usr/本地/sbin/xrdp，咁 xrdp 就安裝得正確。一般嚟講，如果喺安裝之前冇對源碼作出任何變更，配置檔案 xrdp.ini 就會儲存喺 /etc/rxdp。使用 root 權限，用 vim 或者其他編輯器打開佢。
sudo vim /etc/xrdp/xrdp.ini
We need to change some of the configurations to make xrdp run correctly on macOS.
我哋需要修改一啲配置項來令到 xrdp 喺 macOS 上面正確運行。
Find [vnc-any] line. The following lines are below:
搵到 [vnc-any] 呢行。呢行下面幾行嘅內容：
[vnc-any]
name=vnc-any
lib=libvnc.so
ip=ask
port=5900
username=ask
password=ask
#pamusername=asksame
#pampassword=asksame
#pamsessionmng=127.0.0.1
#delay_ms=2000
We need to make some changes, like below.
修改一啲部分，如下。
[vnc-any]
name=vnc-any
lib=libvnc.dylib
ip=127.0.0.1
port=5900
username=ask
password=ask
#pamusername=asksame
#pampassword=asksame
#pamsessionmng=127.0.0.1
#delay_ms=2000
lib file should be libvnc.dylib on macOS instead of libvnc.so on Linux. ip should be set to 127.0.0.1 by default.
喺 macOS 上面，lib 文件應該為 libvnc.dylib，而唔係 Linux 上嘅 libvnc.so。設定默認嘅 ip 為 127.0.0.1。
There are other session types, just comment them if not used, or they may cause some remote desktop client cannot connect successfully. Some clients may not display the xrdp portal correctly and lead to a crash.
佢仲有其他嘅 session 種類，如果唔使用嘅話刪除或者註釋咗佢即可。不然嘅話佢哋可能導致喺某些遠程桌面客戶端上面嘅連接失敗。一啲客戶端可能顯示唔到 xrdp 嘅 portal 介面，並導致崩潰。
After editting the configuration file, save changes and exit vim.
編輯完配置文件之後，保存並且退出 vim。
Use root to run xrdp.
sudo /usr/local/sbin/xrdp
sudo /usr/local/sbin/xrdp-sesman
Or, if the directory is in the PATH,
或者，如果個目錄喺 PATH 之中嘅話，
sudo xrdp
sudo xrdp-sesman
I use Remote Desktop Manager to connect Nucintosh remotely. After I added a RDP session, it connected successfully. xrdp works. However, I soon found that I still cannot adjust the resolution. It is a limitation from macOS, not from ARD.
使用 Remote Desktop Manager 遠程連接 NUC。添加一個 RDP session 之後，佢成功連接。xrdp 工作正常。但係我好快發現佢仍然唔可以係客戶端調整分辨率。呢個係系統嘅限制，而唔係 ARD 嘅限制。
xrdp can work on macOS, making it possible to use RDP (including the built-in Remote Desktop on Windows) connect to Mac. RDP cannot realize the "dynamic resolution" on macOS and some Linux distros. These limitations are from the operating systems, not from remote desktop protocols.
xrdp 可以喺 macOS 上面運行，可以令到 macOS 使用 RDP 來被遠程連接。RDP 喺 macOS 及一啲 Linux 發行版本上面唔可以實現「動態分辨率」功能。呢啲限制係由於系統，並非遠程桌面協議控制。
I create this article because of the annoying confusing problems I met in the Python code. I have to admit that Python is a awesome programming language however… it DOSE exist some weird problems when coding not carefully. So this passage is used to record the problems and solutions for Python debugging.
Well, here's the problem. It happened when I was completing my tutorials. The tutorial imported a Python package called lucent, which is used to visualize the layers of CNNs. The code was like this:
from lucent.optvis import render
import lucent.modelzoo.util as lu_zoo
It seems normal, but it may run into trouble because I will run this code on Jupyter Notebook (local or online). Then I use the lucent package to visualize the layers.
model1.to(ldevice).eval()
lu_zoo.get_model_layers(model1)
model1_conv_images = []
model1_conv_names = [
"conv5:0", "conv5:1", "conv5:3", "conv5:7",
"conv5:15", "conv5:31", "conv5:63", "conv5:127"
]
lu_zoo.get_model_layers(model1)
for x in model1_conv_names:
out = render.render_vis(model1, x, show_image=False)
model1_conv_images.append(np.squeeze(out[0]))
plt.figure(figsize=(8, 4))
show_imgs(model1_conv_images, nc=4)
Then the output cell run into trouble. There should be a beautiful progress bar loading in the output cell showing the current progress of the analyzation, however, the progress bar flashed and disappeared suddenly, leaving some static time text in the output cell. Like the cell below.
5it/s
second
second
second
The situation is not what we want.
The solution is simple too. Now the tqdm package provides a special module for Jupyter Notebook, so we can use that module to show progress bar in Jupyter Notebook environment. However, the tqdm package is not import directly and it is import by another package. How can we replace the original tqdm method to the new specified Jupyter Notebook tqdm method?
Change the code file in the lucent package and make it use notebook.tqdm directly DOES work… but it seems that it is not that "normal". (i.e. totally a strange method, does not robust and stable at all). So, any other methods?
Luckily, there is an elegant way to solve this problem. We import sys the system package first. Then we import the tqdm.notebook package. Finally we can "patch" the tqdm with tqdm.notebook.tqdm, and this will help us enable the Jupyter Notebook progress bar.
import sys
from tqdm import notebook
sys.modules["tqdm"] = notebook
Re-run the Jupyter Notebook, and we find that everything goes well.
Also, this method works well with other situations when some packages need to be change in certain imported packages.
本文原作者未知，轉載自網絡，僅供參考。
我要到伊犁採一朵大大的棉花，
我要到成都砍一根長長的竹子，
我要做一根世界上最大最長的棉簽，
我要在午夜之前給月亮做一個核酸，
明天晚上，
月亮的光，
將照亮我們門上的封條。
]]>