Skip to content

Updates to spectral utility functions for use in rear irradiance decomposition #576

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cdeline merged 9 commits into from
Nov 5, 2025
Merged

Updates to spectral utility functions for use in rear irradiance decomposition #576

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e157096
Add spectrum integration function, updated weather file handling
mjprilliman May 30, 2025
2a3ddb5
Update calls to generate spectra, including adding inputs for min and
mjprilliman Sep 17, 2025
d158a07
Merge branch 'development' into spectral-fixes
mjprilliman Sep 17, 2025
0766739
Check for minute column in weather data (i.e. calculate at 30 minutes
mjprilliman Oct 29, 2025
bef88cc
Merge branch 'development' into spectral-fixes
mjprilliman Oct 30, 2025
18c01b4
Revert change in readWeatherFile outputs (must be metdata, not tuple
mjprilliman Oct 31, 2025
62cf9db
Add test_integrated_spectrum in test_spectra.py.
cdeline Oct 31, 2025
161215c
inline comment cleanup
cdeline Oct 31, 2025
939747a
Merge pull request #1 from cdeline/mjprilliman-spectral-fixes
mjprilliman Nov 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 41 additions & 8 deletions bifacial_radiance/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -978,7 +978,7 @@ def readWeatherFile(self, weatherFile=None, starttime=None,
coerce_year=coerce_year, label=label,
tz_convert_val=tz_convert_val)

return self.metdata
return self.metdata, metadata
Copy link
Contributor

@cdeline cdeline Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mjprilliman - you need to revert this line - other functions are expecting a br.MetObj return here, not a tuple. If you need other raw values from metadata, let's include it in the MetObj, but most of the relevant metadata details should already be included directly accessible in the MetObj. Maybe we can include an additional MetObj attribute like metadata_raw that stores the original raw metadata?

Copy link
Collaborator Author

@mjprilliman mjprilliman Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that was an artifact from a previous version of this effort, and now the metadata handling is done in pvdeg and passed to the MetObj. So reverting this is no problem.



def readWeatherData(self, metadata, metdata, starttime=None,
Expand Down Expand Up @@ -1088,7 +1088,7 @@ def _correctMetaKeys(m):
# put correct keys on m = metadata dict

m['altitude'] = _firstlist([m.get('altitude'), m.get('elevation')])
m['TZ'] = _firstlist([m.get('TZ'), m.get('Time Zone'), m.get('timezone')])
m['TZ'] = _firstlist([m.get('TZ'), m.get('Time Zone'), m.get('timezone'), m.get('tz')])

if not m.get('city'):
try:
Expand Down Expand Up @@ -3149,12 +3149,13 @@ def _printRow(analysisobj, key):


def generate_spectra(self, metdata=None, simulation_path=None, ground_material=None, scale_spectra=False,
scale_albedo=False, scale_albedo_nonspectral_sim=False, scale_upper_bound=2500):
scale_albedo=False, scale_albedo_nonspectral_sim=False, scale_upper_bound=2500, min_wavelength=280, max_wavelength=4000):

"""
Generate spectral irradiance files for spectral simulations using pySMARTS
Or
Generate an hourly albedo weighted by pySMARTS spectral irradiances

#
Parameters
----------
metdata : radianceObject.metdata, optional
Expand Down Expand Up @@ -3212,14 +3213,15 @@ def generate_spectra(self, metdata=None, simulation_path=None, ground_material=N
scale_spectra=scale_spectra,
scale_albedo=scale_albedo,
scale_albedo_nonspectral_sim=scale_albedo_nonspectral_sim,
scale_upper_bound=scale_upper_bound)
scale_upper_bound=scale_upper_bound,
min_wavelength=min_wavelength, max_wavelength=max_wavelength)

if scale_albedo_nonspectral_sim:
self.metdata.albedo = weighted_alb.values
return (spectral_alb, spectral_dni, spectral_dhi, weighted_alb)

def generate_spectral_tmys(self, wavelengths, weather_file, location_name, spectra_folder=None,
output_folder=None):
def generate_spectral_tmys(self, wavelengths, location_name, spectra_folder=None,
output_folder=None, source="TMY"):
"""
Generate a series of TMY-like files with per-wavelength irradiance. There will be one file
per wavelength. These are necessary to run a spectral simulation with gencumsky
Expand Down Expand Up @@ -3247,10 +3249,41 @@ def generate_spectral_tmys(self, wavelengths, weather_file, location_name, spect
output_folder = os.path.join('data','spectral_tmys')
if not os.path.exists(output_folder):
os.makedirs(output_folder, exist_ok=True)



su.generate_spectral_tmys(wavelengths=wavelengths, spectra_folder=spectra_folder,
weather_file=weather_file, location_name=location_name,
metdata=self.metdata, location_name=location_name,
output_folder=output_folder)

def integrated_spectrum(self, spectra_folder):
Copy link
Contributor

@cdeline cdeline Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mjprilliman - we should come up with another pytest in test_spectra.py that exercise this integrated_spectrum function. I have one on my local computer that can be added to development once this gets merged...

Copy link
Contributor

@cdeline cdeline Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm ok with merging this into development if you think it's good to go.

"""
Generate integrated sum of spectrum from SMARTS generated spectra for use in normalization equations

Paramters:
----------
weather_file: (path or str)
File path or path-like string pointing to the weather file used for spectra generation.
The structure of this file, and it's meta-data, will be copied into the new files.
spectra_folder: (path or str)
File path or path-like string pointing to the folder contained the SMARTS generated spectra

Returns:
--------
spectrum: dict
Dictionary with the integrated spectrum sums for DNI, DHI, DNI-albedo product, and DHI-albedo-product
"""
from bifacial_radiance import spectral_utils as su

if spectra_folder is None:
spectra_folder = 'spectra'



spectrum = su.integrated_spectrum(spectra_folder=spectra_folder,
metdata=self.metdata)
return spectrum


# End RadianceObj definition

Expand Down
115 changes: 98 additions & 17 deletions bifacial_radiance/spectral_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from scipy import integrate
from tqdm import tqdm
from pvlib import iotools
from bifacial_radiance import main as main


class spectral_property(object):
Expand Down Expand Up @@ -237,15 +238,15 @@ def spectral_albedo_smarts_SRRL(YEAR, MONTH, DAY, HOUR, ZONE,


def generate_spectra(metdata, simulation_path, ground_material='Gravel', spectra_folder=None, scale_spectra=False,
scale_albedo=False, scale_albedo_nonspectral_sim=False, scale_upper_bound=2500):
scale_albedo=False, scale_albedo_nonspectral_sim=False, scale_upper_bound=2500, min_wavelength=280, max_wavelength=4000):
"""
generate spectral curve for particular material. Requires pySMARTS

Parameters
----------
metdata : bifacial_radiance MetObj
DESCRIPTION.
simulation_path: bifacial_radiance MetObj
MetObj containing weather data, with a datetime index.
simulation_path: string or path
path of simulation directory
ground_material : string, optional
type of ground material for spectral simulation. Options include:
Expand Down Expand Up @@ -295,22 +296,25 @@ def generate_spectra(metdata, simulation_path, ground_material='Gravel', spectra
dni = metdata.dni[idx]
dhi = metdata.dhi[idx]
ghi = metdata.ghi[idx]
alb = metdata.albedo[idx]
if metdata.albedo is not None:
alb = metdata.albedo[idx]
else:
alb = 0.2
solpos = metdata.solpos.iloc[idx]
zen = float(solpos.zenith)
azm = float(solpos.azimuth) - 180
lat = metdata.latitude

# create file names
suffix = f'_{str(dt.year)[-2:]}_{dt.month:02}_{dt.day:02}_{dt.hour:02}.txt'
suffix = f'_{str(dt.year)[-2:]}_{dt.month:02}_{dt.day:02}_{dt.hour:02}_{dt.minute:02}.txt'
dni_file = os.path.join(simulation_path, spectra_folder, "dni"+suffix)
dhi_file = os.path.join(simulation_path, spectra_folder, "dhi"+suffix)
ghi_file = os.path.join(simulation_path, spectra_folder, "ghi"+suffix)
alb_file = os.path.join(simulation_path, spectra_folder, "alb"+suffix)

# generate the base spectra
try:
spectral_dni, spectral_dhi, spectral_ghi = spectral_irradiance_smarts(zen, azm, min_wavelength=280)
spectral_dni, spectral_dhi, spectral_ghi = spectral_irradiance_smarts(zen, azm, min_wavelength=min_wavelength, max_wavelength=max_wavelength)
except:
if scale_albedo_nonspectral_sim:
walb[dt] = 0.0
Expand Down Expand Up @@ -389,7 +393,7 @@ def generate_spectra(metdata, simulation_path, ground_material='Gravel', spectra

return (spectral_alb, spectral_dni, spectral_dhi, None)

def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_name, output_folder):
def generate_spectral_tmys(wavelengths, spectra_folder, metdata, location_name, output_folder):
"""
Generate a series of TMY-like files with per-wavelength irradiance. There will be one file per
wavelength. These are necessary to run a spectral simulation with gencumsky
Expand All @@ -400,8 +404,8 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n
array or list of integer wavelengths to simulate, in units [nm]. example: [300,325,350]
spectra_folder: (path or str)
File path or path-like string pointing to the folder contained the SMARTS generated spectra
weather_file: (path or str)
File path or path-like string pointing to the weather file used for spectra generation
metdata: pandas DataFrame
DataFrame containing the weather data, with a datetime index.
location_name:
_description_
output_folder:
Expand All @@ -413,8 +417,10 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n
spectra_files.sort()

# -- read in the weather file and format
(tmydata, metdata) = iotools.read_tmy3(weather_file, coerce_year=2021)
tmydata.index = tmydata.index+pd.Timedelta(hours=1)
#(tmydata, metdata) = main.RadianceObj.readWeatherFile(weatherFile=weather_file, coerce_year=2021)
#(tmydata, metdata) = iotools.read_tmy3(weather_file, coerce_year=2021)
tmydata = metdata.tmydata.copy()
#tmydata.index = tmydata.index+pd.Timedelta(hours=1)
tmydata.rename(columns={'dni':'DNI',
'dhi':'DHI',
'temp_air':'DryBulb',
Expand All @@ -426,8 +432,9 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n
dtindex = tmydata.index

# -- grab the weather file header to reproduce location meta-data
with open(weather_file, 'r') as wf:
header = wf.readline()
# with open(weather_file, 'r') as wf:
# header = wf.readline()
header = metdata.metadata.copy()

# -- read in a spectra file to copy wavelength-index
temp = pd.read_csv(os.path.join(spectra_folder,spectra_files[0]), header=1, index_col = 0)
Expand All @@ -438,7 +445,7 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n
take = file[4:-4]
if take not in dates:
dates.append(take)
dates = pd.to_datetime(dates,format='%y_%m_%d_%H').tz_localize(dtindex.tz)
dates = pd.to_datetime(dates,format='%y_%m_%d_%H_%M').tz_localize(dtindex.tz)

# -- create a multi-index of columns [timeindex:alb,dni,dhi,ghi]
iterables = [dates,['ALB','DHI','DNI','GHI']]
Expand All @@ -449,7 +456,7 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n

# -- fill with irradiance data
for file in spectra_files:
a = pd.to_datetime(file[4:-4],format='%y_%m_%d_%H').tz_localize(dtindex.tz)
a = pd.to_datetime(file[4:-4],format='%y_%m_%d_%H_%M').tz_localize(dtindex.tz)
b = file[:3].upper()
spectra_df[a,b] = pd.read_csv(os.path.join(spectra_folder,file),header=1, index_col=0)

Expand All @@ -476,8 +483,82 @@ def generate_spectral_tmys(wavelengths, spectra_folder, weather_file, location_n
wave_df.loc[col[0],col[1]] = spectra_df[col].loc[wave]

with open(fileName, 'w', newline='') as ict:
for line in header:
ict.write(line)
# for line in header:
# ict.write(line)
wave_df.to_csv(ict, index=False)


def integrated_spectrum(spectra_folder, metdata ):
"""
Generate integrated sums across the full spectra

Paramters:
----------
spectra_folder: (path or str)
File path or path-like string pointing to the folder contained the SMARTS generated spectra
metdata: pandas DataFrame
DataFrame containing the weather data, with a datetime index.


Returns:
-------
integrated_sums: (list)
list of integrated sums for DNI, DHI, DNI*ALB, DHI*ALB
"""

# -- read in the spectra files
spectra_files = next(os.walk(spectra_folder))[2]
spectra_files.sort()

# -- read in the weather file and format
#(tmydata, metdata) = main.RadianceObj.readWeatherFile(weatherFile=weather_file, coerce_year=2021)
#(tmydata, metdata) = iotools.read_tmy3(weather_file, coerce_year=2021)
tmydata = metdata.tmydata.copy()
#tmydata.index = tmydata.index+pd.Timedelta(hours=1)
tmydata.rename(columns={'dni':'DNI',
'dhi':'DHI',
'temp_air':'DryBulb',
'wind_speed':'Wspd',
'ghi':'GHI',
'relative_humidity':'RH',
'albedo':'Alb'
}, inplace=True)
dtindex = tmydata.index

# -- grab the weather file header to reproduce location meta-data
# with open(weather_file, 'r') as wf:
# header = wf.readline()
header = metdata.metadata.copy()

# -- read in a spectra file to copy wavelength-index
temp = pd.read_csv(os.path.join(spectra_folder,spectra_files[0]), header=1, index_col = 0)

# -- copy and reproduce the datetime index
dates = []
for file in spectra_files:
take = file[4:-4]
if take not in dates:
dates.append(take)
dates = pd.to_datetime(dates,format='%y_%m_%d_%H_%M').tz_localize(dtindex.tz)

# -- create a multi-index of columns [timeindex:alb,dni,dhi,ghi]
iterables = [dates,['ALB','DHI','DNI','GHI']]
multi_index = pd.MultiIndex.from_product(iterables, names=['time_index','irr_type'])

# -- create empty dataframe
spectra_df = pd.DataFrame(index=temp.index,columns=multi_index)
# -- fill with irradiance data
for file in spectra_files:
a = pd.to_datetime(file[4:-4],format='%y_%m_%d_%H_%M').tz_localize(dtindex.tz)
b = file[:3].upper()
spectra_df[a,b] = pd.read_csv(os.path.join(spectra_folder,file),header=1, index_col=0)
integrated_sums = pd.DataFrame(index=dates, columns=['Sum_DNI', 'Sum_DHI', 'Sum_DNI_ALB', 'Sum_DHI_ALB'])
for col in spectra_df.columns:
integrated_sums.loc[col[0], 'Sum_DNI'] = integrate.trapezoid(spectra_df[col[0], 'DNI'], spectra_df.index)
integrated_sums.loc[col[0], 'Sum_DHI'] = integrate.trapezoid(spectra_df[col[0], 'DHI'], spectra_df.index)
integrated_sums.loc[col[0], 'Sum_DNI_ALB'] = integrate.trapezoid(spectra_df[col[0], 'DNI'] * spectra_df[col[0], 'ALB'], spectra_df.index)
integrated_sums.loc[col[0], 'Sum_DHI_ALB'] = integrate.trapezoid(spectra_df[col[0], 'DHI'] * spectra_df[col[0], 'ALB'], spectra_df.index)

return integrated_sums


Loading