AI-Enabled Reinforcement Learning Framework Using Deep Q-Networks for Intelligent Energy Management in Nano-Grids Supporting Sensitive Medical Equipment

 

 

 

 

 

 

3.3 Action-Space Encoding

Within RL, DQN in particular, the choices available to agents are shaped by what is in the action space. Energy management at the nano-grid level requires that all possible control strategies open to the energy management system (EMS) at a point in time be part of the action space. These tasks require governing the movement of electricity between various components on the nano-grid, consisting of battery storage, photovoltaic (PV) system, local loads and the outside power grid. Specifying the action space accurately helps the RL agent learn how to manage economic performance, system stability and battery health together. Unlike DDPG and PPO which keep actions continuous, the DQN algorithm only works with discrete action spaces. Therefore, each control strategy must produce a distinct command that can be directly interpreted by the nano-grid. After thorough analysis of practical energy flow configurations and typical smart inverter functionalities, five discrete actions were designed and encoded:

: Grid Supply Only (No Battery or PV Involvement)

When this takes place, the nano-grid uses the external grid alone to cover the full power demand. The main reason for this feature is when no solar power is available and the battery’s charge is insufficient to discharge it. Even though this isn’t the most economical thing to do, it is necessary to maintain power for people and to avoid outages. It is used when none of the regular energy routes are available because the system operates under certain limits.

: Battery Charging (up to 2 kW)

When the PV system produces more energy than the home uses or when electricity rates are low, it sends the surplus toward charging the battery. Charging occurs only when the battery is not fully charged (below 95%) to stop overcharging. Doing this allows companies to make the most of price differences in the future and increases the system’s independence by keeping electricity for peak demand hours. Normally, the charging rate is set to 2 kW because going beyond that may cause the battery to heat up and possibly suffer damage.

Battery Charging Limit:

Where  is the maximum allowed state-of-charge

: Battery Discharging (up to 2 kW)

In this way, the battery releases energy to cover energy shortages in a particular area. Batteries will only allow to discharge while the SOC is above 20% to preserve the batteries. This means the home saves energy from the grid, mostly when it is priced highly or when it makes little PV energy. The ability to discharge energy can prevent sudden drop in available power or stops during a rise in demand. In the same way, discharge power is controlled by the limits of the hardware and the safety threshold.

: PV to Load and Battery

Here, solar energy is used by the area it is produced in, instead of being offered back to the grid. If there’s spillover energy, it is used to charge the battery. It maximizes how much renewable energy we use and reduces the need to interact with the electrical grid. In areas with plenty of sunshine, it brings great benefits and is also encouraged by special schemes in many jurisdictions. The agent learns, in cases of high solar power and moderate load, to focus on this action to make sure PV energy is well captured and saved.

: Grid Export

The battery is used only if excess solar power is left over after local buildings’ needs are met and the battery is fully charged or is not charging at that time. If there’s surplus energy, it is sent back to the grid. Feed-in tariffs or net metering schemes make it necessary to do this in dynamic pricing environments. Selling energy abroad helps the agent make use of extra electricity and supports stable power supply on the grid. It only becomes an option after every other way of consuming or storing gas is abused.

Energy Export to Grid:

Where  is the surplus energy sent to the grid.

Design Considerations and Constraints

It is not possible to choose or define each action without considering the physical, economic and safety limitations in the system. To ensure realistic simulation and deployment, several feasibility checks are implemented:

Battery Charging/Discharging Constraints: Actions  and  are gated by SOC thresholds. If an action is selected but violates the battery's safety margins (e.g., attempting to charge when SOC ≥ 95%), the agent is penalized through the reward function or forced to default to .

Battery SoC Update:

Where  is the state of charge at time  are charging/discharging energy,  are charging/discharging efficiency and  is the maximum energy capacity.

PV Availability:Only in the presence of solar generation are actions  and  allowed. There would be no benefit and all the previous steps would be pointless if we attempted to export energy  at night or when solar radiation is zero.

Load Availability:When there is no load, some actions are unnecessary. If the battery is tried to be discharged when the car doesn’t need electricity, the energy used becomes less efficient and results in penalties.Encoding each movement as a separate variable shrinks the output layer of the network, making the Q-learning challenge simpler. In DQNs, every output node links to one of the five actions and its value means how much the action is worth in the current situation. While learning from the Bellman equation, the Q-values are kept up to date and the policy soon chooses the action with the most resources.

Action-Space Scalability and Extensions

Scalability and ease of future upgrades are significant strengths of using the proposed action-space structure. Since the present model provides one fixed power level, we can make action spaces more flexible by stepwise changing the levels. For instance:

·       Charging at 1 kW, 1.5 kW, or 2 kW

·       Discharging in similar gradations

·       Introducing time-aware actions like "delay load" or "precharge battery for evening peak"

In addition, the agent may control controllable loads such as EVs, HVAC units and smart appliances, so it decides both the source of power and the pace at which they consume it. The new methods require agents to move from single-action choices to managing a variety of actions which could be achieved by relying on advanced actor-critic approaches or combining RL approaches into one system. When there are many energy sources or batteries involved, the action space can be widened to include tasks where both a source and a battery are managed at the same moment. Because the number of learning actions is large in these configurations, learning is best done by relying on methods that organize problems.

 

Real-World Implications of Action-Space Design

Adopting the five-action model is in step with the system architecture of today’s residential and commercial microgrids. Most of these commercial products allow API-level control of charge, discharge and grid communication, so adding our decision engine to the system is very straightforward. In addition, when every action is mapped to a clear physical operation, it becomes easier to understand the behavior of the RL agent which is important for energy systems. Therefore, if the agent usually picks A₃ during days when the sun is out, it is clear to stakeholders that self-consumption is a priority for the system.

 

3.4 Reward Function Formulation

The reward function is very important for any reinforcement learning algorithm, showing the agent what behaviors to avoid or embrace. For nano-grid energy management, the reward system has to take into account the conflicting targets of cost efficiency, reliable energy supply and ensuring the equipment lives long.At each timestep , the scalar reward  is computed as:

Where  is the monetary cost of energy consumed from the grid at time  calculated as ,  is the battery degradation penalty, estimated by a quadratic function of change in SOC: , capturing nonlinear wear effects, and  is the load penalty term for unmet energy demand. A penalty of 1 cent per unit of load not met by the nano-grid is calculated if they cannot meet the required demand. We have tuned the coefficients  via grid search to the values . Compared to those three factors, emphasis is placed mainly on reducing costs. To improve learning that lasts, we use a future reward value factor of , helping decisions that are beneficial but happen in the long run. Not discharging the battery deeply today may improve the energy freedom tomorrow. Gradients are kept controlled by limiting the range of the reward to [−10, 1]. Each episode is normalized separately to make seasonal datasets more alike. To test its strength, the reward structure was evaluated in situations of blackouts, battery failures and negative pricing events caused by an abundance of renewable energy. As a result, the agent improves its strategy to preserve battery SOC when energy is costly and seeks chances to use cheap energy for charging.

Grid Import Calculation:

Where  is the energy imported from the grid, and  is the battery discharge at time .

Cost of Grid Energy:

Where  is the cost at time  and  is the electricity price (real-time/ToU).

Battery Degradation Penalty:

Where  is the change in state-of-charge and  is the time increment.

Load Not Served Penalty

Where  is the unserved demand at time .

 

3.5 Deep Q-Network (DQN) Architecture Design

DQN is central to the proposed reinforcement learning-based energy management system and it instantaneously turns large amounts of data into smart control decisions. Because nano-grid energy systems are nonlinear, stochastic and involve multiple constraints, the complexity prevents the use of classical optimization or rules-based solutions. Instead, DeepMind introduced DQN, a value-based reinforcement learning algorithm that helps learning from information available through experience and layout deep neural networks to tackle large tasks. The primary goal of a DQN is to find the optimal action-value function,  which shows the expected overall reward an agent can get after taking action  in state  and following the best strategy. The DQN uses a neural network as Q-function approximator because it becomes too difficult to compute or keep Q-values for every possible combination in big or continuous state spaces. This network’s parameters which are often labeled , are adjusted step by step through temporal-difference learning and several versions of the Bellman equation. To achieve its goal of fast inference and stable learning, the proposed DQN is realized as a fully connected feedforward neural network. Every component is scaled between 0 and 1 through min-max normalization so large features do not affect the results and updates are smooth.

 

Input Layer:

The input layer is made up of 4 neurons which stand for the original state variables after normalization. It provides information to the hidden layers without doing anything else to it, except for normalizing it.

Hidden Layer 1:

The layer under the input layer is the first hidden layer and involves 128 neurons, each using the Rectified Linear Unit (ReLU) function. With this layer, the network can learn about the complex connection made between electricity price and how much power is sent from the battery.

Hidden Layer 2:

The next layer below the input layer is called the first hidden layer and each of its 128 neurons runs the ReLU function. Because of this layer, the network can grasp the complex ways electricity price is affected by the amount of power delivered from the battery.

Output Layer:

The output layer contains 5 neurons, corresponding to the 5 discrete actions defined in the action space: . The Q-value produced by an output neuron is  which estimates the reward  associated with action  and state . At execution time, the agent chooses the option with the greatest Q-value. Best architecture was found by measuring various model depths and widths to balance convergence speed, computational requirements and final performance. A simple network failed to include every interaction, but a notable overfitting problem appeared with the early use of more intricate networks.

 

Training Mechanism and Optimization

To train the DQN, the loss function is defined as the Mean Squared Error (MSE) between the predicted Q-value and the target Q-value derived from the Bellman update:

Where the target value  is calculated as:

Here,  is the target Q-network with parameters , and  is the discount factor, which governs the agent’s consideration of future rewards. One key feature in DQN is having a different target network that is updated regularly to the main network every 100 steps. During training, the Adam optimizer with a learning rate of 0.0005 is used. Adam is selected because its adjustable learning rate makes training converge swiftly at the start and helps prevent the model from overshooting afterwards. For 1500 episodes, training simulates an entire day every time, with each interval representing a 15-minute decision-making step using 96 steps. Backpropagation is used to update the weights after each small group of samples is used.

An experience replay buffer with a capacity of 10,000 transitions is used in the DQN to aid learning and reduce connection between successive data. After everything in each environment step, the tuple  is put into the buffer. During training, samples of 64 are taken randomly from the buffer at once. The method increases stability since the model experiences more variety in how it progresses between states in each update. With experience replay, offline training in future is possible, since the model trains on synthetic or old nano-grid data, making learning and data acquisition different steps.

Where  is the trainable network weights,  is the batch size, and  is the target Q-value for sample .

Stabilization Techniques and Regularization

Additionally, deep learning algorithms often apply other strategies besides experience replay and target nets to control and stabilize the learning process.

·       Reward Clipping: Since extreme levels of reward can cause the learning to fluctuate and explode in gradient descent, any reward outside the range [-10, 1] is replaced by one of these two values.

·       Early Stopping and Model Checkpoints: The rate of improvement is measured with validation reward curves. Provided that the reward metric is still low, training is called off after 100 episodes. These weights are later found by check pointing.

·       Exploration Strategy (ε-Greedy): The agent acts based on ε-greediness, randomly choosing an acton with probability ε and otherwise taking the action with the greatest predicted Q-value. Over 1000 episodes, the ε-value is lowered from 1.0 to 0.05 exponentially so that exploration and exploitation switch gradually.

·       TensorBoard Monitoring: TensorBoard is used to save and inspect training loss curves, Q-value charts and reward progress.

Epsilon-Greedy Policy:

Where  is the number of actions.

 

Inference and Edge Deployment

After training converges, the final model is frozen and changed into TensorFlow Lite format. By doing this, the model can be used on resource-limited devices like Raspberry Pi 4 and Jetson Nano, where it is needed for continuous control at the edges. Testing the model on a Raspberry Pi showed that it requires less than 40 milliseconds per step which more than meets the requirement to make decisions in less than 15 minutes.In nano-grid energy management, the suggested DQN system ensures the framework is robust by including experience replay, target network synchronization, reward shaping and inference optimization. Its simple architecture allows it to accurately and flexibly handle the varying and complex environment found in nano-grid energy. Because it continues to learn from real situations, the DQN goes beyond mere rules and provides a modern and active approach to energy systems, centered on efficiency, sustainability, autonomy and economic feasibility.

Input:   Normalized Environment States:

              Action Space:

              Learning Parameters:

              Time Horizon: (episodes × timesteps)

              Replay Buffer Capacity: , Batch Size:

Output:Optimized Q-Network

              Trained policy

Step 1: Initialize

   Initialize Q-network with random weights

   Initialize target network with weights

   Initialize Experience Replay Buffer

   Set exploration rate

Step 2: For each episode to :

   Reset environment and obtain initial state

   For each time step  to

Step 3: Action Selection (ε-greedy policy)

  

Step 4: Apply Action and Update Environment

   If

        

   If

        

                                                                 // Update Battery SoC

Step 5: Compute Grid Exchange and Energy

Balances

  

  

Step 6: Compute Reward

                                                                                  // Grid Cost

                                                                                             // Battery Degradation

                                     // Unserved Load

                                                               // Total Reward

   Clip reward:

Step 7: Store Transition

   Store into replay buffer

Step 8: Sample and Train

   If buffer has  transitions:

         Sample  tuples

                                                                     // Compute Target Q-values

                                                            // Minimize Loss (MSE)

   Backpropagate gradients and update weights

Step 9: Target Network Update

   Every  steps, update

Step 10: Decay

  

Step 11: End Episode

   Save model if performance improved

   Log training metrics (reward, loss) using TensorBoard

Return:

   Trained Q-network

   Optimal Policy

The framework was developed in a Python 3.11 environment using important libraries such as TensorFlow 2.13, NumPy, Pandas and Matplotlib to conduct artificial intelligence, data management and visualization tasks related to energy management. All experiments were run on a Windows 11 Pro 64-bit system with an Intel Core i7-12700H CPU, 16 GB RAM and NVIDIA RTX 3050 Laptop GPU, to speed up the training. Using specially created logic, the environment simulation was set up with actual data from NREL and Pecan Street for both real-time load and PV generation. Training involved 1500 episodes and each of these lasted 96 steps (representing 15 minutes each in the simulation). During training, the DQN used the Adam algorithm and an exploration strategy where epsilon was slowly being reduced. Learning stability was achieved by saving experience and updating the network target in parallel. When training finished, the model was changed to TensorFlowLite format and tested on a Raspberry Pi 4B (4GB RAM) to confirm that it can inference quickly in edge environments, making the model ready for practical use in nano-grids.

Table 2 Energy Cost (USD/day) vs Grid Dependency (%)

In Table 2 and Figure 3, energy management strategies are evaluated using the two key indicators Energy Cost (USD/day) and Grid Dependency (%). These measures allow us to judge the effectiveness of energy management systems (EMS) in both smart grid and distributed energy networks.It is clear from the results that the Proposed DQN (Deep Q-Network) model performs best, with an energy cost of 5.14 USD/day and a dependency on the grid of just 56.73%. Therefore, a DQN controller is both energy efficient and increases the independence of the system from the main grid.

Figure 3. Grid Dependency by Model

The Rule-Based and SOC-Threshold methods use the most energy, about 7.34 and 7.18 USD each day and are highly connected to the grid, with both exceeding 85%. Large-scale processes usually fail to respond to shifts in needs, resulting in unnecessary costs.The use of Fuzzy Logic EMS, Heuristic EMS and GA-Scheduler all show moderate results in lessening both cost and reliance on resource management. Using MPC-Linear, DNN-Controller and LSTM-Agent allows for bigger gains since these use prediction and learning.From all the models, Hybrid-RL and Proposed DQN perform the best in terms of cutting expenses and keeping the system independent. DQN is shown to be superior to Hybrid-RL on both factors, confirming its strong and flexible design.The main point from Table 2 is that using reinforcement learning and particularly DQN, is very effective for handling energy management in an optimal, cost-reducing and independent way.

Table 3 Renewable Utilization (%) vs Battery Degradation Index

Table 3and Figure 4 compares several energy models based on their usage of renewables and the loss of battery function over time. The Proposed DQN model clearly shows the greatest renewable use and lowest battery deterioration, making it superior to any other approach. The findings show that using DQN as the strategy increases the use of green energy and keeps batteries running smoothly. The two conventional techniques—Rule-Based and SOC- Threshold—had degraded renewable use of 62.31% and 66.92% and degradation indices of 0.46 and 0.42. Because these degradation values are higher, they could lead to battery cycling that shortens the battery’s life in continuous deployment.

Figure 4. Renewable Utilization by Model

The results lead to more effective control strategies, in turn making the crucial metrics improve. Illustratively, both Fuzzy Logic EMS and Heuristic EMS raise renewable usage above 75% while lowering battery damage to around 0.31–0.33. It is shown that methods such as MPC-Linear, DNN-Controller and LSTM-Agent confirm that as models become more complex, their performance rises. Prior to DQN, the Hybrid-RL model achieves high usage of renewables and a low degradation level which means it performs well. Across all the experiments, the Proposed DQN performs the best, demonstrating its suitability for environmentally friendly, renewable and battery-supported energy management.

Table 4 System Efficiency (%) vs Load Served (%)

Table 4and Figure 5 shows a comparison of different energy management models by looking at System Efficiency (%) and Load Served (%). Results from the data show both metrics have consistently improved as AI replaces traditional and rule-based methods in the models. The system under Rule-Based starts with an efficiency of 71.91% and load served of 88.06%, showing that it does not react well to dynamic energy changes. The SOC-Threshold and Fuzzy Logic EMS models achieved 75.06% and 77.43% efficiency which resulted in a similar increase in power supplied to the loads. Using state-of-charge data with fuzzy logic improves the way these models help with deciding what to do next.

Figure 5. System Efficiency and Load Served by Model

Improvements are also seen in Heuristic EMS and GA-Scheduler which rely on optimization approaches and heuristic methods to handle operational plans. Models help the company achieve 81% efficiency and handle more than 93% of the load. These models make use of predictive control and deep learning which help them quickly change their actions as load and generation change. Because it can process temporal data, the LSTM-Agent accomplishes 86.17% efficiency and 96.38% load served. The Hybrid-RL model and the Proposed DQN offer the finest performance of the team’s approaches. Using reinforcement learning allows these approaches to overtake others strongly and the Proposed DQN reaches peak efficiency of 91.93% and provides 98.83% of the system’s needed power. As a result, deep reinforcement learning is particularly useful for smart energy systems since it excels in learning and decision making.

 

Loss of Power Supply Probability (LPSP) and Q-Value Convergence Rate (Steps) are used to compare energy management models, as shown in Table 5 and Figure 6. LPSP measures the percentage of total demand not met owing to energy shortages, but the Q-value convergence rate shows us how effectively an RL model picks out the best policy. Better model performance is shown by a lower score for both metrics. Because they cannot learn and rely partly on rules, the Rule-Based and SOC-Threshold models have LPSP values of 4.84% and 4.29% but fail to converge fast or at all, so they are only used in this research as baselines.

 

Table 5 LPSP (%) vs Q-Value Convergence Rate (Steps)

 

Figure 6. Q-Value Convergence Rate vs Model

With the change to smart controllers Fuzzy Logic EMS and Heuristic EMS, LPSP drops somewhat to 3.91% and 3.62% respectively and requires fewer steps to converge.Results with GA-Scheduler and MPC-Linear optimization showed better LPSP values of 3.25% and 2.71%, showing a stronger dependable system. Raven Edition shows the strongest advancements with RL-based models. With deep learning helping direct policy choices, the DNN-Controller and LSTM-Agent show LPSPs of 2.45% and 2.18% respectively and the number of steps needed for convergence drops to 628 and 580.Both the Hybrid-RL and the Proposed DQN demonstrate the finest performance by reducing the LPSP to 1.92% and 1.34%, respectively and reaching convergence at 508 and 439 steps. It shows that deep reinforcement learning is effective for balancing the reliability of the system with fast learning. As a result, the Proposed DQN is a powerful and reliable energy management tool.

 

In Table 6 and Figure 7, will find a comparison of various energy management techniques by energy cost (expressed as USD/day) and battery degradation index. Running the Rule-Based model consumes the most energy every day ($7.38) and causes a high level of battery degradation (0.44) which is expensive and results in quicker wear. Both the SOC-Threshold and Fuzzy Logic EMS methods achieve minor enhancements, decreasing energy cost to $7.00 and $6.73 and degradation to 0.41 and 0.38, through improved control routines. Both the Heuristic EMS and GA-Scheduler show improvements by using the battery better, reducing expenses to $6.41 and $6.10 and decreasing degradation indices to 0.34 and 0.31.

 

Table 6 Energy Cost (USD/day) vs Battery Degradation Index

In Table 6 and Figure 7, will find a comparison of various energy management techniques by energy cost (expressed as USD/day) and battery degradation index. Running the Rule-Based model consumes the most energy every day ($7.38) and causes a high level of battery degradation (0.44) which is expensive and results in quicker wear. Both the SOC-Threshold and Fuzzy Logic EMS methods achieve minor enhancements, decreasing energy cost to $7.00 and $6.73 and degradation to 0.41 and 0.38, through improved control routines. Both the Heuristic EMS and GA-Scheduler show improvements by using the battery better, reducing expenses to $6.41 and $6.10 and decreasing degradation indices to 0.34 and 0.31.

Figure 7.Energy Cost and Battery Degradation Index by Control Strategy

Smart predictive optimization allows the MPC-Linear model to achieve minimum degradation (0.28) and pay $5.89 for energy.There are a wide range of outcomes shown by deep learning models. The DNN-Controller shows a decrease in performance deficit (0.26), even as it requires more energy ($6.26). Long-term learning in the LSTM-Agent explains why it did better with these results, having performed with greater accuracy at $5.920 and lower variance at 0.22. Hybrid-RL and the DQN we proposed worked the best. The Hybrid-RL model has a good balance at $5.71 per day and 0.20 as a degradation index, closely trailed by the Proposed DQN which performs best with energy cost of $5.19 and only 0.16 as battery degradation.

Table 7 Grid Dependency (%) vs System Efficiency (%)

Table 7and Figure 8 shows how various energy management strategies perform when evaluated according to Grid Dependence and System Efficiency. The degree to which a system depends on external electricity is shown by Grid Dependency, while System Efficiency explains how well it uses available resources which can be clean energy and stores. These traditional models, Rule-Based and SOC-Threshold, both depend strongly on the grid, showing grid dependency levels of 88.15% and 83.62%, respectively. At the same time, they have low system efficiencies of 72.64% and 75.33%. Because they depend on established control rules, these models find it difficult to adapt and maximize efficiency rapidly.After using more complex systems, including Fuzzy Logic EMS and Heuristic EMS, mild grid dependence is observed (79.03% and 76.41%) together with very good gains in efficiency (77.26% and 79.88%). When GA-Scheduler and MPC-Linear optimization methods are used, grid dependence is reduced by 6.78% and 6.16%, respectively and the computer’s efficiency rises to 81.27% and 83.41%.

Figure 8. Grid Dependency vs System Efficiency by Model

Deep learning is successfully used in both the DNN-Controller and LSTM-Agent for predictive control, resulting in better benefits of 67.95% and 64.78% lower dependency and 84.39% and 86.41% higher efficiency. Both the Proposed DQN and Hybrid-RL which use reinforcement learning, achieved the best results in our experiments. The Proposed DQN shows it can handle energy use more autonomously than other methods, as it has the highest efficiency and lowest grid dependency.

Table 8 Renewable Utilization (%) vs Load Served (%)

The Renewable Utilization (%), as well as the Load Served (%) for each energy management model, are shown in Table 8 and Figure 9 and both play a crucial role in judging the smart grid system’s sustainability and dependability. The results demonstrate that a better performance on both metrics can be seen as models get smarter and more complex.As the basic Rule-Based model has the least outcome, it offers 61.18% renewable utilization and serves 86.03% of the total load. As a result, Nigerians are greatly dependent on the grid and find it difficult to provide enough energy. The SOC-Threshold and Fuzzy Logic EMS approaches perform slightly better, leading to increases in renewable use to 66.09% and 70.37% and 88.45% and 90.24% of total served load for each.As a result of heuristic methods like Heuristic EMS, GA-Scheduler and MPC-Linear, there has been a sustained increase in trend. These models use energy intelligently, allowing renewables to power around 76% of demand and serve more than 94% of total need.

Figure 9. Renewable Utilization vs Load Served by Model

The DNN-Controller and LSTM-Agent models led to faster system speed and improved predictions, raising utilization to 80% and serving 96.15% of the requested load.The models Hybrid-RL and the Proposed DQN show the power of reinforcement learning in getting the highest performance. In particular, the Proposed DQN exceeds 94% for renewable energy and nearly 99% for load coverage.

Table 9 LPSP (%) vs Energy Cost (USD/day)

Table 9and Figure 10 shows the results for various energy management approaches, measured by LPSP % and daily energy currency. Reliability, as shown by LPSP, means that the system provided less than 100% of the energy needed, whereas the control strategy’s economic efficiency is shown by its energy cost. The purpose of any advanced energy management system is to lower power consumption and waste. Static and reactive, Rule-Based leads to both the highest LPSP (4.79%) and daily energy cost ($7.29).

Figure 10. LPSP vs Energy Cost by Model

This means it can’t respond to changes in demand or use energy wisely, so it often ends up using expensive electric grid power or not delivering power when needed. The SOC-Threshold and Fuzzy Logic EMS models achieve slightly better results by adding condition-based logic and doing some fuzzy reasoning, lowering LPSP to 4.25% and 3.88% and reducing daily energy costs to $7.01 and $6.58. With Heuristic EMS, GA-Scheduler and MPC-Linear, more power can be balanced effectively. The use of these models sees LPSP fall below 3% and energy costs drop to $6.00–$5.77, proving smarter use of resources. Modeling with DNN-Controller and LSTM-Agent uses knowledge from the past and leads to improved operation, cutting LPSP to 2.55% and 2.19% and reducing energy usage. The top results are achieved with Hybrid-RL and Proposed DQN which are both based on reinforcement learning. With 1.32% in LPSP and a daily cost of $5.13, the Proposed DQN is the clear winner for excellent energy optimization and reliable use of resources.

 

The DQN-powered energy system was tested comparatively against nine well-known models on eight different performance measures, all using data from real-world loads and solar panels. There is strong evidence that operations are now more efficient, cost less and energy is supplied more reliably. The DQN performed best when it came to cutting costs for energy, averaging 5.13 USD a day versus 5.89 USD for MPC, 6.10 USD for GA-

 

Scheduler and 5.71 USD from the tested hybrid approaches. The reason costs dropped was because the model learned to secure low-price energy during the day and distribute it back to the grid when demand was high. Furthermore, the DQN showed great improvement in renewable energy use, reaching a rate of 94.28%, far better than the 82.94% from the nearest baseline. As a result, the PV system is used more and less energy is drawn from the grid than usual.

The technical results show that the proposed system has a system efficiency rating of 91.93% and a battery degradation index of 0.16. It shows that the DQN works well for energy and also supports the battery by teaching itself to charge and discharge smoothly. In addition, the model performed excellently by reaching a Load Served Percentage of 98.83% which is better than heuristic EMS, MPC and even LSTM-based agents. The low LPSP of 1.32% recorded by the model confirmed that it supports balanced supply and demand of power in a variety of environmental circumstances.The algorithm demonstrated solid learning consistency and reached convergence easily. Compared to many other models, CIN reaches convergence in Q-values in 439 steps, much sooner than most models that typically take 500–900 steps. Since the system rapidly converged, it shows that the reward system was well-suited to match its goals and the neural network design was able to represent the different relationships between load, PV, electricity pricing and batteries.

 

The results from the DQN-based agent proved that the algorithm can generalize well to new irradiance and load patterns, achieving similar metrics with only little changes (below 2%). It suggests that the model can work well in practical situations where some uncertainties cannot be avoided completely. In essence, the discussion suggests that the proposed DQN outperforms others in learning control policies that are smart, handy and maintain their value over time. With its fast response and versatile on-device construction, it may be used to develop AI-driven energy systems for use in rural microgrids, smart buildings and off-grid renewable systems.

Future Work

This study confirms that DQN vastly improves the ability and efficiency of energy management systems in nano-grids powered by AI. Treating the environment as a Markov Decision Process and processing real-time data including the battery level, demand, solar generation and electricity prices, the proposed agent finds the best way to adjust power between storage, solar and the grid. The proposed system is evaluated against eight important metrics and is shown to do better than rule-based, optimization-based and hybrid reinforcement learning methods without fail. It achieved a minimum cost of 5.13 USD a day, used renewable energy 94.28% of the time, suffered little battery degradation (only 0.16) and lost power supply just 1.32% of the time, making it the top choice among the three groups. It will be useful to consider electric vehicles, demand-side management and community energy sharing in future work by including these in the action and state spaces. Connecting energy coordination efforts to federated learning and blockchain can support security, a decentralized system and privacy protection. Furthermore, applying this framework to multi-agent reinforcement learning (MARL) may help manage energy collectively in diverse energy systems. Because the DQN-based energy manager generalizes smoothly and copes with real-time events, it forms an excellent base for applying self-adaptive control systems in the expanding field of renewable energy and smart grids.

 

1.     S. A. G. K. Abadi, T. Khalili, S. I. Habibi, A. Bidram, and J. M. Guerrero, "Adaptive control and management of multiple nano-grids in an islanded dc microgrid system," IET Gener. Transm. Distrib., vol. 17, no. 8, pp. 1799–1815, 2023, doi: 10.1049/gtd2.12556.

2.     S. A. G. K. Abadi and A. Bidram, "Effective utilization of grid-forming cloud hybrid energy storage systems in islanded clustered dc nano-grids for improving transient voltage quality and battery lifetime," IET Gener. Transm. Distrib., vol. 17, no. 8, pp. 1836–1856, 2023, doi: 10.1049/gtd2.12775.

3.     M. Tofighi-Milani, S. Fattaheian-Dehkordi, M. Fotuhi-Firuzabad, and M. Lehtonen, "Distributed reactive power management in multi-agent energy systems considering voltage profile improvement," IET Gener. Transm. Distrib., vol. 17, no. 21, pp. 4891–4906, 2023, doi: 10.1049/gtd2.13005.

4.     R. Krishna and H. S, "Long short-term memory-based forecasting of uncertain parameters in an islanded hybrid microgrid and its energy management using improved grey wolf optimization algorithm," IET Renew. Power Gener., vol. 18, no. 16, pp. 3640–3658, 2024, doi: 10.1049/rpg2.13115.

5.     S. Yong et al., "Environmental Self-Adaptive Wind Energy Harvesting Technology for Self-Powered System by Triboelectric-Electromagnetic Hybridized Nanogenerator with Dual-Channel Power Management Topology," Adv. Energy Mater., vol. 12, no. 43, p. 2202469, 2022, doi: 10.1002/aenm.202202469.

6.     S. Wankhede and L. Kamble, "Performance investigation of electric vehicle battery thermal management system using nano fluids as coolants on ANSYS CFX software," Energy Storage, vol. 5, no. 4, p. e420, 2023, doi: 10.1002/est2.420.

7.     M. Wahidujjaman et al., "Enhanced Stability and Performance of Islanded DC Microgrid Systems Using Optimized Fractional Order Controller and Advanced Energy Management," Engineering Reports, vol. 7, no. 4, p. e70122, 2025, doi: 10.1002/eng2.70122.

8.     U. Srivastava and R. R. Sahoo, "Discharging Performance Analysis of MXene Nano-Enhanced Phase Change Material for Double and Triplex Tube Thermal Energy Storage," Energy Storage, vol. 6, no. 7, p. e70055, 2024, doi: 10.1002/est2.70055.

9.     S. Tan et al., "Achieving Broadband Microwave Shielding, Thermal Management, and Smart Window in Energy-Efficient Buildings," Adv. Funct. Mater., vol. 35, no. 8, p. 2415921, 2025, doi: 10.1002/adfm.202415921.

10.   M. H. Tahir et al., "Optimizing Electrical Efficiency and Levelized Cost of Energy in Photovoltaic Systems Through Thermal Management Using Microchannel Heat Sinks," Int'l Jnl of Energy Research, vol. 2025, no. 1, p. 2433429, 2025, doi: 10.1155/er/2433429.

11.   G. Chen et al., "The potential application of the triboelectric nanogenerator in the new type futuristic power grid intelligent sensing," EcoMat, vol. 5, no. 11, p. e12410, 2023, doi: 10.1002/eom2.12410.

12.   E. Tunçbilek et al., "Impact of nano-enhanced phase change material on thermal performance of building envelope and energy consumption," Int J Energy Res, vol. 46, no. 14, pp. 20249–20264, 2022, doi: 10.1002/er.8200.

13.   Islam et al., "Exploring the Thermal Potential of Shape Stabilized Graphene Nano Platelets Enhanced Phase Change Material for Thermal Energy Storage," Energy Technol., vol. n/a, no. n/a, p. 2400337, 2024, doi: 10.1002/ente.202400337.

14.   F. Ghojavand et al., "Simulation and performance analysis of a cooling energy storage system based on encapsulated nano-enhanced phase change materials," Energy Storage, vol. 5, no. 4, p. e424, 2023, doi: 10.1002/est2.424.

15.   Yazdaninejadi and H. Ebrahimi, "A new protection algorithm for tackling the impact of fault-resistance and cloud energy storage on coordination of recloser-fuse protection," IET Gener. Transm. Distrib., vol. 17, no. 8, pp. 1827–1835, 2023, doi: 10.1049/gtd2.12691.

16.   Y. Hu et al., "Nano-Metal-Organic Frameworks and Nano-Covalent-Organic Frameworks: Controllable Synthesis and Applications," Chem. Asian J., vol. 20, no. 1, p. e202400896, 2025, doi: 10.1002/asia.202400896.

17.   B. Zhang et al., "Functionalized Bone Implant Inspired by Lattice Defense Strategy: Grid Management, Precise and Effective Multiple-Prevention of Osteomyelitis Recurrence and Promote Bone Regeneration," Adv. Healthcare Mater., vol. 14, no. 6, p. 2403058, 2025, doi: 10.1002/adhm.202403058.

18.   B. Chen and Z. L. Wang, "Toward a New Era of Sustainable Energy: Advanced Triboelectric Nanogenerator for Harvesting High Entropy Energy," Small, vol. 18, no. 43, p. 2107034, 2022, doi: 10.1002/smll.202107034.

19.   F. R. Badal et al., "Microgrid to smart grid's evolution: Technical challenges, current solutions, and future scopes," Energy SciEng, vol. 11, no. 2, pp. 874–928, 2023, doi: 10.1002/ese3.1319.

20.   Rayan et al., "An Efficient Energy Management Routing and Scalable Topology in Wireless Sensor Network Using Virtual Backbone," Wireless Communications, vol. 2022, no. 1, p. 9327318, 2022, doi: 10.1155/2022/9327318.

21.   Y. Zhan et al., "Micro-Nano Structure Functionalized Perovskite Optoelectronics: From Structure Functionalities to Device Applications," Adv. Funct. Mater., vol. 32, no. 24, p. 2200385, 2022, doi: 10.1002/adfm.202200385.

22.   C. Boobalan and S. K. Kannaiyan, "A correlation to predict the thermal conductivity of MXene-silicone oil based nano-fluids and data driven modeling using artificial neural networks," Int J Energy Res, vol. 46, no. 15, pp. 21538–21547, 2022, doi: 10.1002/er.7786.

23.   J. Cofré-Toledo et al., "Numerical prediction of the solidification and melting of encapsulated nano-enhanced phase change materials," Energy Storage, vol. 6, no. 1, p. e521, 2024, doi: 10.1002/est2.521.

24.   J. C. Das and D. De, "Nano-scale design of full adder and full subtractor using reversible logic based decoder circuit in quantum-dot cellular automata," Int J Numer Model, vol. 36, no. 5, p. e3092, 2023, doi: 10.1002/jnm.3092.

25.   M. M. Kamal, I. Ashraf, and E. Fernandez, "Planning and optimization of standalone microgrid with renewable resources and energy storage," Energy Storage, vol. 5, no. 1, p. e395, 2023, doi: 10.1002/est2.395.