Skip to content

GitLab

Commit 85716f2a authored by Chanelle Lee's avatar Chanelle Lee
Browse files

Added querying so starts and stops automatically and efficiently with no... 

Added querying so starts and stops automatically and efficiently with no wasted time. Have also redone testParams using new experimental model
parent 847fa433
Expand all Hide whitespace changes
Inline Side-by-side
.ipynb_checkpoints/ProbConfusionAnalysis-checkpoint.ipynb
View file @ 85716f2a
This diff is collapsed.
This diff is collapsed.
.ipynb_checkpoints/ProbConfusionAnalysis_dep-checkpoint.ipynb 0 → 100755
View file @ 85716f2a
This diff is collapsed.
This diff is collapsed.
ProbConfusionAnalysis.ipynb
View file @ 85716f2a
This diff is collapsed.
This diff is collapsed.
ProbConfusionAnalysis_dep.ipynb
View file @ 85716f2a
......@@ -1229,29 +1229,6 @@
" for i, sigma in enumerate(sigmas)])"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"scrolled": true
},
"outputs": [
{
"ename": "NameError",
"evalue": "name 'testParams_DF' is not defined",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-11-d89ffc317e6f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtestParams_DF\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mNameError\u001b[0m: name 'testParams_DF' is not defined"
]
}
],
"source": [
"print(testParams_DF)"
]
},
{
"cell_type": "code",
"execution_count": null,
......
%% Cell type:code id: tags:
``` python
import numpy as np
import scipy
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
```
%% Cell type:markdown id: tags:
# Analysis of the probability of confusing qualities for different values of sigma
%% Cell type:markdown id: tags:
First need the pdf and cdf of the normal distributions which will represent the noise around each of the quality values
%% Cell type:code id: tags:
``` python
def f(x, mu, s):
return scipy.stats.norm.pdf(x, loc=mu, scale=s)
```
%% Cell type:code id: tags:
``` python
def F(x, mu, s):
return scipy.stats.norm.cdf(x, loc=mu, scale=s)
```
%% Cell type:markdown id: tags:
Qualities are set in the interval (0,1) and equally spaced
%% Cell type:code id: tags:
``` python
n = 5
qualities = {i: i/(n+1) for i in range(1, (n+1))}
```
%% Cell type:code id: tags:
``` python
def integrand(x, i, j, s):
return f(x, qualities[i], s)*F(x, qualities[j], s)
def probConfusion(i, j, s):
return scipy.integrate.quad(integrand, -np.inf, np.inf, args=(i, j, s))
```
%% Cell type:code id: tags:
``` python
s0 = 0.0000001
sf = 100
ss = np.linspace(s0, sf, 1000)
confusions = [probConfusion(1, 2, s)[0] for s in ss]
```
%% Cell type:code id: tags:
``` python
plt.plot(ss, confusions)
```
%%%% Output: execute_result
[<matplotlib.lines.Line2D at 0x22e0f4ad240>]
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
ss = np.linspace(0.0000001, 10, 1000)
confusions = [probConfusion(1, 2, s)[0] for s in ss]
plt.plot(ss, confusions)
```
%%%% Output: execute_result
[<matplotlib.lines.Line2D at 0x22e0f548f28>]
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
ss = np.linspace(0.0000001, 10, 1000)
confusions = [[probConfusion(1, j, s)[0] for s in ss] for j in [2, 3, 4, 5]]
for confusion in confusions:
plt.plot(ss, confusion)
```
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
ss = np.linspace(0.0000001, 10, 1000)
confusions = [[probConfusion(1, j, s)[0] for s in ss] for j in [2, 3, 4, 5]]
for confusion in confusions:
plt.plot(ss, confusion)
```
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
ss = np.linspace(0.0000001, 5, 1000)
confusions = [[probConfusion(1, j, s)[0] for s in ss] for j in [2, 3, 4, 5]]
for confusion in confusions:
plt.plot(ss, confusion)
```
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
d = {'sigma':ss, **{'q{}'.format(j): confusions[i] for i, j in enumerate([2, 3, 4, 5])}}
```
%% Cell type:code id: tags:
``` python
df = pd.DataFrame(data=d)
```
%% Cell type:code id: tags:
``` python
df.head()
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
0 1.000000e-07 0.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00
1 5.005105e-03 4.104751e-122 0.000000e+00 0.000000e+00 0.000000e+00
2 1.001011e-02 1.051201e-31 3.577518e-122 7.451264e-274 0.000000e+00
3 1.501511e-02 5.734691e-15 2.733803e-55 9.343211e-123 2.336542e-219
4 2.002012e-02 4.075678e-09 6.984412e-32 7.149700e-70 3.085665e-124
%% Cell type:markdown id: tags:
### Here want to get some idea of appropriate sigma values - how about checking the cut offs for having at least a 10% chance of confusion for all sites within a range:
%% Cell type:code id: tags:
``` python
testParams_DF = pd.DataFrame(columns=['sigma', 'alpha', 'note'])
```
%% Cell type:markdown id: tags:
Range is whole quality space
%% Cell type:code id: tags:
``` python
df.loc[(df['q2'] >= 0.1) & (df['q3'] >= 0.1) & (df['q4'] >= 0.1) & (df['q5'] >= 0.1)].head()
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
74 0.370370 0.375167 0.262259 0.169892 0.101546
75 0.375375 0.376777 0.265031 0.173131 0.104590
76 0.380380 0.378347 0.267745 0.176322 0.107618
77 0.385385 0.379878 0.270401 0.179466 0.110627
78 0.390390 0.381372 0.273002 0.182563 0.113616
%% Cell type:markdown id: tags:
$\sigma = 0.37$
- q2 : 0.38
- q3 : 0.26
- q4 : 0.17
- q5 : 0.10
%% Cell type:markdown id: tags:
Range is q4 or closer
%% Cell type:code id: tags:
``` python
df.loc[(df['q2'] >= 0.1) & (df['q3'] >= 0.1) & (df['q4'] >= 0.1)].head()
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
56 0.280280 0.337069 0.200187 0.103577 0.046294
57 0.285285 0.339767 0.204346 0.107618 0.049227
58 0.290290 0.342380 0.208409 0.111625 0.052198
59 0.295295 0.344911 0.212380 0.115597 0.055202
60 0.300300 0.347365 0.216260 0.119531 0.058234
%% Cell type:markdown id: tags:
$\sigma=0.28$
- q2 : 0.34
- q3 : 0.20
- q4 : 0.10
- q5 : 0.04
%% Cell type:markdown id: tags:
Range is q3 or closer
%% Cell type:code id: tags:
``` python
df.loc[(df['q2'] >= 0.1) & (df['q3'] >= 0.1)].head()
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
37 0.185185 0.262259 0.101546 0.028119 0.005455
38 0.190190 0.267745 0.107618 0.031517 0.006595
39 0.195195 0.273002 0.113616 0.035049 0.007867
40 0.200200 0.278043 0.119531 0.038698 0.009270
41 0.205205 0.282880 0.125357 0.042451 0.010803
%% Cell type:markdown id: tags:
$\sigma = 0.19$
- q2 : 0.26
- q3 : 0.10
- q4 : 0.03
- q5 : 0.01
%% Cell type:markdown id: tags:
Range is just q2
%% Cell type:code id: tags:
``` python
df.loc[(df['q2'] >= 0.1)].head()
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
19 0.095095 0.107618 0.006595 0.000100 3.576083e-07
20 0.100100 0.119532 0.009270 0.000206 1.242683e-06
21 0.105105 0.131087 0.012463 0.000384 3.644447e-06
22 0.110110 0.142242 0.016153 0.000662 9.294145e-06
23 0.115115 0.152973 0.020303 0.001066 2.110231e-05
%% Cell type:markdown id: tags:
$\sigma = 0.1$
- q2 : 0.11
- q3 : 0.01
- q4 : 0.00
- q5 : 0.00
%% Cell type:code id: tags:
``` python
sns.lineplot(data=df.set_index('sigma'))
```
%%%% Output: execute_result
<matplotlib.axes._subplots.AxesSubplot at 0x22e106ee1d0>
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
sns.lineplot(data=df.where(df['sigma'] <= 1.0).set_index('sigma'))
# plotting the values of sigma
plt.axvline(0.1, alpha=0.3, color='grey')
plt.axvline(0.19, alpha=0.3, color='grey')
plt.axvline(0.28, alpha=0.3, color='grey')
plt.axvline(0.37, alpha=0.3, color='grey')
# plotting P = 0.1
plt.axhline(0.1, alpha=0.3, color='grey')
```
%%%% Output: execute_result
<matplotlib.lines.Line2D at 0x22e1083ae80>
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
desProb = 0.1
range_all = df[(df['q2'] >= desProb) & (df['q3'] >= desProb) & (df['q4'] >= desProb) & (df['q5'] >= desProb)].iloc[0]
range_q4 = df[(df['q2'] >= desProb) & (df['q3'] >= desProb) & (df['q4'] >= desProb)].iloc[0]
range_q3 = df[(df['q2'] >= desProb) & (df['q3'] >= desProb)].iloc[0]
range_q2 = df[(df['q2'] >= desProb)].iloc[0]
df_1Ranges = pd.DataFrame(data = [range_all, range_q4, range_q3, range_q2])
```
%% Cell type:code id: tags:
``` python
df_1Ranges
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
74 0.370370 0.375167 0.262259 0.169892 1.015460e-01
56 0.280280 0.337069 0.200187 0.103577 4.629378e-02
37 0.185185 0.262259 0.101546 0.028119 5.454770e-03
19 0.095095 0.107618 0.006595 0.000100 3.576083e-07
%% Cell type:code id: tags:
``` python
testParams_DF = testParams_DF.append([{'sigma': row[1], 'alpha': row[2], 'chance': '10%', 'note': 'Prob_q2'} for row in df_1Ranges.itertuples()], sort=True)
```
%% Cell type:code id: tags:
``` python
alphas_1 = [np.mean(row[2:]) for row in df_1Ranges.itertuples()]
alphas_1
```
%%%% Output: execute_result
[0.22721601881269943,
0.17178190793707013,
0.09934474874734342,
0.028578442773621412]
%% Cell type:code id: tags:
``` python
testParams_DF = testParams_DF.append([{'sigma': row[1], 'alpha': alphas_1[i], 'chance': '10%', 'note': 'mean'}
for i, row in enumerate(df_1Ranges.itertuples())])
```
%% Cell type:markdown id: tags:
### Here want to get some idea of appropriate sigma values - how about checking the cut offs for having at least a 25% chance of confusion for all sites within a range:
%% Cell type:code id: tags:
``` python
desProb = 0.25
range_all = df[(df['q2'] >= desProb) & (df['q3'] >= desProb) & (df['q4'] >= desProb) & (df['q5'] >= desProb)].iloc[0]
range_q4 = df[(df['q2'] >= desProb) & (df['q3'] >= desProb) & (df['q4'] >= desProb)].iloc[0]
range_q3 = df[(df['q2'] >= desProb) & (df['q3'] >= desProb)].iloc[0]
range_q2 = df[(df['q2'] >= desProb)].iloc[0]
df_25Ranges = pd.DataFrame(data = [range_all, range_q4, range_q3, range_q2])
```
%% Cell type:code id: tags:
``` python
df_25Ranges
```
%%%% Output: execute_result
sigma q2 q3 q4 q5
140 0.700701 0.433217 0.368292 0.306930 0.250550
105 0.525526 0.411280 0.326893 0.250550 0.184855
70 0.350350 0.368292 0.250550 0.156453 0.089228
35 0.175175 0.250550 0.089228 0.021781 0.003561
%% Cell type:code id: tags:
``` python
sns.lineplot(data=df.where(df['sigma'] <= 1.0).set_index('sigma'))
# plotting the values of sigma
plt.axvline(df_25Ranges['sigma'].iloc[0], alpha=0.3, color='grey')
plt.axvline(df_25Ranges['sigma'].iloc[1], alpha=0.3, color='grey')
plt.axvline(df_25Ranges['sigma'].iloc[2], alpha=0.3, color='grey')
plt.axvline(df_25Ranges['sigma'].iloc[3], alpha=0.3, color='grey')
# plotting P = 0.25
plt.axhline(desProb, alpha=0.3, color='grey')
```
%%%% Output: execute_result
<matplotlib.lines.Line2D at 0x22e11174400>
%%%% Output: display_data
[Hidden Image Output]
%% Cell type:code id: tags:
``` python
testParams_DF = testParams_DF.append([{'sigma': row[1], 'alpha': row[2], 'chance': '25%', 'note': 'Prob_q2'} for row in df_25Ranges.itertuples()])
```
%% Cell type:code id: tags:
``` python
alphas_25 = [np.mean(row[2:]) for row in df_25Ranges.itertuples()]
```
%% Cell type:code id: tags:
``` python
testParams_DF = testParams_DF.append([{'sigma': row[1], 'alpha': alphas_25[i], 'chance': '25%', 'note': 'mean'}
for i, row in enumerate(df_25Ranges.itertuples())])
```
%% Cell type:code id: tags:
``` python
```
%% Cell type:code id: tags:
``` python
sigmas = set(testParams_DF['sigma'])
chances = [testParams_DF.loc[testParams_DF['sigma'] == sigma, 'chance'].iloc[0] for sigma in sigmas]
testParams_DF = testParams_DF.append([{'sigma': sigma, 'alpha': 0.1, 'chance': chances[i], 'note': 'control'}
for i, sigma in enumerate(sigmas)])
```
%% Cell type:code id: tags:
``` python
print(testParams_DF)
```
%%%% Output: error
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-11-d89ffc317e6f> in <module>()
----> 1 print(testParams_DF)
NameError: name 'testParams_DF' is not defined
%% Cell type:code id: tags:
``` python
testParams_DF['paramSet'] = range(1, len(testParams_DF)+1)
testParams_DF = testParams_DF.set_index('paramSet')
print(testParams_DF)
```
%% Cell type:code id: tags:
``` python
testParams_DF.to_csv('testParams.csv')
```
%% Cell type:code id: tags:
``` python
testParams_DF.iloc[0]['sigma']
```
%% Cell type:code id: tags:
``` python
set(testParams_DF.index.values)
```
%% Cell type:code id: tags:
``` python
testParams_DF.index.values
```
%% Cell type:code id: tags:
``` python
testParams_DF.iloc[15]['sigma']
```
%% Cell type:code id: tags:
``` python
def convert_paramSetNum(filePath):
df = pd.read_csv(filePath, names=['index', 'time', 'paramSet', 'simNum', 'timestep', 'agent', 'iteration', 'option', 'belief'])
df = df.drop('index', axis=1)
df.to_csv(filePath, header=False, index=False)
```
%% Cell type:code id: tags:
``` python
import os, glob
path = 'Results/'
all_files = glob.glob(os.path.join(path, "botSim_beliefs_[0-4].csv"))
for f in all_files:
convert_paramSetNum(f)
```
%% Cell type:code id: tags:
``` python
```
......
botSimMain.py
View file @ 85716f2a
......@@ -21,13 +21,36 @@ def main(numSims=10):
else:
logger.critical('Could not connect to streaming server')
time.sleep(5)
# setup signal for checking finish
vrep.simxSetIntegerSignal(clientID,
'asdf',
1,
vrep.simx_opmode_blocking)
testParams = pd.read_csv('testParams.csv', index_col='paramSet')
for paramNum in set(testParams.index.values):
if paramNum < 17: # This just allows us to skip the ones already tested
continue
# if paramNum < 17: # This just allows us to skip ones already tested
# continue
print('Param num: {}'.format(paramNum))
simNum = 0
for i in range(0, numSims):
# IMPORTANT
# you should poll the server state to make sure
# the simulation completely stops before starting a new one
notStopped = True
while notStopped:
# poll the useless signal (to receive a message from server)
vrep.simxGetIntegerSignal(clientID,
'asdf',
vrep.simx_opmode_blocking)
# check server state (within the received message)
res, info = vrep.simxGetInMessageInfo(
clientID,
vrep.simx_headeroffset_server_state)
# check bit0
notStopped = info & 1
print('Not Stopped yet')
simNum += 1
print('Sim num: {}'.format(simNum))
sretCode = vrep.simxStartSimulation(clientID,
......@@ -52,7 +75,8 @@ def main(numSims=10):
logger.critical('Simulation not finished'
+ ' properly! {}'.format(eretCode))
sys.exit('Could not finish simulation {}'.format(eretCode))
time.sleep(240)
vrep.simxFinish(clientID)
def start(paramNum, simNum, noise, distrust):
......
botSimTODOS.todo
View file @ 85716f2a
Code:
☐ Investigate why botSimMain not logging to the console?
☐ Add ability to query for a simulation to have stopped, so can speed up transition between simulations.
✔ Add ability to query for a simulation to have stopped, so can speed up transition between simulations. @done (19-04-01 15:52)
☐ Find out the cause of the Unicode Decode Error in simxCallScriptFunction
✔ Add exception capture in broadcasting thread @done (19-04-01 13:42)
Experiments:
✔ Need to finish ParamSet 11 (3-10) @done (19-03-27 11:43)
_1 logs contain paramSets 2-7 complete
_2 logs contain paramsets 8-10 complete and 11 partial (1-2)
_3 logs contain paramsets 12-15 complete
_4 logs contain paramset 16 complete
_5 logs contain paramset 1 complete
_6 logs contain paramset 11 partial (3-10)
__
Analysis:
myThreads.py
View file @ 85716f2a
......@@ -103,7 +103,11 @@ class BroadcastingThread(threading.Thread):
while not self.stopped():
# Message is set from the ePuck side allowing for msg changes
# without having to stop and start Broacasting Threads
self.ePuck.sendMessage(self.ePuck.msgToSend,
self.messageName)
time.sleep(random.random())
try:
self.ePuck.sendMessage(self.ePuck.msgToSend,
self.messageName)
time.sleep(random.random())
except UnicodeDecodeError:
logger.warning('Unicode error captured with msg'
'{}'.format(self.ePuck.msgToSend))
logger.debug('BroadcastingThread has finished')
testParams.csv 0 → 100755
View file @ 85716f2a
paramSet,alpha,chance,note,sigma
1,0.0,5%,control,0.0717
2,0.1,5%,control,0.0717
3,0.2,5%,control,0.0717
4,0.3,5%,control,0.0717
5,0.4,5%,control,0.0717
6,0.5,5%,control,0.0717
7,0.05,5%,q2 prob,0.0717
8,0.006328538735734794,5%,mean,0.0717
9,0.0,10%,control,0.092
10,0.1,10%,control,0.092
11,0.2,10%,control,0.092
12,0.3,10%,control,0.092
13,0.4,10%,control,0.092
14,0.5,10%,control,0.092
15,0.1,10%,q2 prob,0.092
16,0.01317041146831397,10%,mean,0.092
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment